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Abstract. We prove a constant term conjecture of Robbins and Zeilberger (J. Com- 
bin. Theory Ser. A 66 (1994), 17-27), by translating the problem into a determinant 
evaluation problem and evaluating the determinant. This determinant generalizes the 
determinant that gives the number of all totally symmetric self-complementary plane 
partitions contained in a (2n) x (2n) x (2n) box and that was used by Andrews (J. Com- 
bin. Theory Ser. A 66 (1994), 28-39) and Andrews and Burge (Pacific J. Math. 158 
(1993), 1-14) to compute this number explicitly. The evaluation of the generalized 
determinant is independent of Andrews and Burge's computations, and therefore in 
particular constitutes a new solution to this famous enumeration problem. We also 
evaluate a related determinant, thus generalizing another determinant identity of An- 
drews and Burge (loc. cit.). By translating some of our determinant identities into 
constant term identities, we obtain several new constant term identities. 



1. Introduction. I started work on this paper originally hoping to find a proof of 
the following conjecture of Robbins and Zeilberger [16, Conjecture C'=B'] (caution: 
in the quotient defining B' it should read {m + 1 + 2j) instead of {m + 1 + j)), which 
we state in an equivalent form. 



1991 Mathematics Subject Classification. Primary 05A15, 15A15; Secondary 05A17, 33C20.. 
Key words and phrases, determinant evaluations, constant term identities, totally symmetric 
self-complementary plane partitions, hypergeometric series. 

^Supported in part by EC's Human Capital and Mobility Program, grant CHRX-CT93-0400 and the 
Austrian Science Foundation FWF, grant P10191-MAT 



1 



Typeset by AmS-T^ 



THE ELECTRONIC JOURNAL OF COMBINATORICS 4 (1997), #RXX 



2 



Conjecture. Let x and n be nonnegative integers. Then 

V no<i<i<n-i(i - ^i^j) nr=o^(i - ^i) J 

' n-1 (n-2)/2 

n (s^StSrSW n (2a; + 2i + l)!(2i)! n is even (1.1a) 

11=0 i=0 
n-1 {n-V)l2 
2" n (3.]S+?;Ki+2.)! n (2^. + 2^)! (2i - l)! ijn is odd. (Lib) 
1=1 1=1 

ifere, CT(Expr) means the constant term in Expr, i.e., the coefficient of z1z2 ■ ■ ■ 
in Expr. 

I thought this might be a rather boring task since in the = there existed 

aheady a proof of the Conjecture (see [16]). This proof consists of translating the 
constant term on the left-hand side of (1.1) into a sum of minors of a particular matrix 
(by a result [16, Corollary D=C] of Zeilberger), which is known to equal the number of 
totally symmetric self- complementary plane partitions contained in a {2n) x {2n) x (2n) 
box (by a result of Doran [4, Theorem 4.1 + Proof 2 of Theorem 5.1]). The number 
of these plane partitions had been calculated by Andrews [1] by transforming the sum 
of minors into a single determinant (using a result of Stembridge [15, Theorem 3.1, 
Theorem 8.3]) and evaluating the determinant. Since Zeilberger shows in [16, Lemma 
D'=C'] that the translation of the constant term in (1.1) into a sum of minors of 
some matrix works for generic x, and since Stembridge's result [15, Theorem 3.1] 
still applies to obtain a single determinant (see (2.2)), my idea was to evaluate this 
determinant by routinely extending Andrews's proof of the totally symmetric self- 
complementary plane partitions conjecture, or the alternative proofs by Andrews and 
Burge [2]. However, it became clear rather quickly that this is not possible (at least 
not routinely). In fact, the aforementioned proofs take advantage of a few remarkable 
coincidences, which break down if x is nonzero. Therefore I had to devise new methods 
and tools to solve the determinant problem in this more general case where x ^ 0. 

In the course of the work on the problem, the subject became more and more 
exciting as I came across an increasing number of interesting determinants that could 
be evaluated, thus generalizing several determinant identities of Andrews and Burge 
[2], which appeared in connection with the enumeration of totally symmetric self- 
complementary plane partitions. In the end, I had found a proof of the Conjecture, 
but also many more interesting results. In this paper, I describe this proof and all 
further results. 

The proof of the Conjecture will be organized as follows. In Theorem 1, item (3) in 
Section 1 it is proved that the constant term in (1.1) equals the positive square root of 
a certain determinant, actually of one determinant, namely (2.2a), if n is even, and of 
another determinant, namely (2.2b), if n is odd. In addition. Theorem 1 provides two 
more equivalent interpretations of the constant term, in particular a combinatorial 
interpretation in terms of shifted plane partitions, which reduces to totally symmetric 
self-complementary plane partitions for x = 0. 
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The main idea that we will use to evaluate the determinants in Theorem 1 will be 
to generalize them by introducing a further parameter, y, see (3.1) and (4.1). The 
generalized determinants reduce to the determinants of Theorem 1 when y = x. Many 
of our arguments do not work without this generalization. In Section 3 we study the 
two-parameter family (3.1) of determinants that contains (2.2a) as special case. If 
y = X + m, with m a fixed integer, Theorem 2 makes it possible to evaluate the 
resulting determinants. This is done for a few cases in Corollary 3, including the case 
y = X (see (3.69)) that we are particularly interested in. Similarly, in Section 4 we 
study the two-parameter family (4.1) of determinants that contains (2.2b) as special 
case. Also here, ii y — x -\- with m a fixed integer. Proposition 5 makes it possible 
to evaluate the resulting determinants. This is done for two cases in Corollary 6, 
including the case y = x (see (4.42)) that we are particularly interested in. This 
concludes the proof of the Conjecture, which thus becomes a theorem. It is restated 
as such in Theorem 11. However, even more is possible for this second family of 
determinants. In Theorem 8, we succeed in evaluating the determinants (4.1) for 
independent x and y, taking advantage of all previous results in Section 4. 

There is another interesting determinant identity, which is related to the afore- 
mentioned determinant identities. This is the subject of Section 5. It generalizes a 
determinant identity of Andrews and Burge [2]. Finally, in Section 6 we translate our 
determinant identities of Sections 4 and 5 into constant term identities which seem to 
be new. Auxiliary results that are needed in the proofs of our Theorems are collected 
in the Appendix. 

Since a first version of this article was written, g-analogues of two of the determi- 
nant evaluations in this article. Theorems 8 and 10, were found in [9]. No g-analogues 
are known for the results in Section 3. Also, it is still open whether the g-analogues of 
[9] have any combinatorial meaning. Another interesting development is that Amde- 
berhan (private communication) observed that Dodgson's determinant formula (see 
[18, 17]) can be used to give a short inductive proof of the determinant evaluation 
in Theorem 10 (and also of its g-analogue in [9]), and could also be used to give an 
inductive proof of the determinant evaluation in Theorem 8 (and its g-analogue in [9] ) 
provided one is able to prove a certain identity featuring three double summations. 

2. Transformation of the Conjecture into a determinant evaluation prob- 
lem. In Theorem 1 below we show that the constant term in (1.1) equals the positive 
square root of some determinant, one if n is even, another if n is odd. Also, we pro- 
vide a combinatorial interpretation of the constant term in terms of shifted plane 
partitions. Recall that a shifted plane partition of shape (Ai, A2, . . . , A,.) is an array 
TT of integers of the form 

TTl,! 7ri,2 7ri,3 TTi^Ai 

7r2,2 7r2,3 7r2,A2 

7I'r,r • • • ^r,Ar 

such that the rows and columns are weakly decreasing. Curiously enough, we need 
this combinatorial interpretation to know that we have to choose the positive root 
once the determinant is evaluated. 
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Theorem 1. Let x and n be nonnegative integers. The constant term in (1.1) equals 

(1) the sum of all nx n minors of the n x (2n — 1) matrix 

J V / 0<i<n-l, 0<j<2n+x-2 

(2) the number of shifted plane partitions of shape {x + n — 1, a; + n — 2, . . . , 1), 
with entries between and n, where the entries in row i are at least n — i, 
i = l,2,...,n-l, 

(3) the positive square root of 

det ( E ifn is even, (2.2a) 

0<i,j<n-l \^+2i-j<r<x+2j-i J 

22- det f (^^+^+^^i)-^^"+;!+"^iy.+^i+t^^^^-^^^ ) tfmsodd, (2.2b) 



if the sums in (2.2a) are interpreted by 



B 

Expr(r) = < 

r=A+l 



B 

J2 Expr(r) A<B 

r=A+l 

A = B (2.3) 

A 

- E Expr(r) A> B. 

r=B+l 



Proof, ad (1). This was proved by Zeilberger [16, Lemma D'=C']. (Note that we 
have performed a shift of the indices i,j in comparison with Zeilberger 's notation.) 

ad (2). Fix a minor of the matrix (2.1), 

0<i,j<n-l \ \Aj — I 

say. By the main theorem of nonintersecting lattice paths [6, Cor. 2; 15, Theorem 1.2] 
(see Proposition Al) this determinant has an interpretation in terms of nonintersect- 
ing lattice paths. By a lattice path we mean a lattice path in the plane consisting 
of unit horizontal and vertical steps in the positive direction. Furthermore, recall 
that a family of paths is called nonintersecting if no two paths of the family have 
a point in common. Now, the above determinant equals the number of all families 
[Pq, Pi, . . . , Pn-i) of nonintersecting lattice paths, where Pi nms from (— 2i,i) to 
(x — Ai, Ai), i = 0, 1, . . . ,n — 1. An example with a; = 2, n = 5, Ai = 1, A2 = 3, A3 = 4, 
A4 = 7, A5 = 9 is displayed in Figure l.a. (Ignore P_i for the moment.) 

Hence, we see that the sum of all minors of the matrix (2.1) equals the number 
of all families (Pq, -Pi, . . . , Pn-i) of nonintersecting lattice paths, where Pi runs from 
(— 2z, i) to some point on the antidiagonal line xi +2:2 = x {xi denoting the horizontal 
coordinate, X2 denoting the vertical coordinate), i = 0, 1, . . . , n — 1. Next, given such 
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a. nonintersecting lattice paths 




c. filling of the regions 

Figure 1 



a family [Pq, Pi, ... , Pn-i) of nonintersecting lattice paths, we shift Pi by the vector 
(z, — z), i = 0, 1, . . . , n— 1. Thus a family {Pq, P{, . . . , -P^_i) of lattice paths is obtained, 
where P/ runs from (— i, 0) to some point on the line xi + X2 — x, see Figure l.b. 
The new paths may touch each other, but they cannot cross each other. Therefore, 
the paths Pq, P{, . . . , P^-i cut the triangle that is bordered by the a;i-axes, the line 
xi + X2 = X, the vertical line xi = — n + 1 into exactly n + 1 regions. We fill these 
regions with integers as is exemplified in Figure I.e. To be more precise, the region to 
the right of Pq is filled with O's, the region between Pq and P{ is filled with I's, ... , 



THE ELECTRONIC JOURNAL OF COMBINATORICS 4 (1997), #RXX 



6 



the region between -P^_2 and P^_i is filled with {n — l)'s, and the region to the left 
of Pn-i is filled with n's. Finally, we forget about the paths and reflect the array of 
integers just obtained in an antidiagonal line, see Figure l.d. Clearly, a shifted plane 
partition of shape + n — 1, x + n — 2, . . . , 1) is obtained. Moreover, it has the desired 
property that the entries in row i are at least n — i, i = l,2,...,n — 1. It is easy to 
see that each step can be reversed, which completes the proof of (2). 

ad (3). It was proved just before that the constant term in (1.1) equals the number 
of all families (Pq, -Pi, • • • , Pn-i) of nonintersecting lattice paths, where Pi runs from 
(— 2z, i) to some point on the antidiagonal line xi + X2 = x, i = 0,1, . . . ,n — 1. 

Now, let first n be even. By a theorem of Stembridge [15, Theorem 3.1] (see 
Proposition A2, with Ai = (— 2z, i), i = 0,1, . . . ,n — 1, I = (the lattice points on the 
line xi + X2 = x)), the number of such families of nonintersecting lattice paths equals 
the Pfaffian 

Pi AQihJ)), (2.4) 

0<i<j<n—l 



where Q{i,j) is the number of all pairs {Pi,Pj) of nonintersecting lattice paths. Pi 
running from {—2i,i) to some point on the line Xi + X2 — x, and Pj running from 
{—2j,j) to some point on the line xi + X2 = x. 

In order to compute the number Q{i,j) for fixed i, j , < i < j < n — 1, we follow 
Stembridge's computation in the proof of Theorem 8.3 in [15]. We define bki to be the 
number of all pairs {Pi, Pj) of intersecting lattice paths, where Pi runs from (— 2z, i) 
to {x — k, k), and where Pj runs from (— 2j, j) to {x — I, I). Since the total number of 
lattice paths from {—2i, i) to xi +X2 — x is 2^+*, it follows that 2^^+*+-^ —Q{h j) is the 
number of pairs of intersecting lattice paths from {—2i, i) and {—2j,j) to xi+X2 = x. 
Hence, 

2^-+^+^ - Q{i, j) ^j2^ki = J2^ik+Yl ^^^^ 

k,l k<l k>l 

the last equality being a consequence of the fact that bki = bik, which is proved by 
the standard path switching argument (find the first meeting point and interchange 
terminal portions of the paths from thereon, see the proofs of [6, Cor. 2; 15, Theo- 
rem 1.2]). When k < I, every path from {—2i, i) to (a; — /, /) must intersect every path 
from {-2j,j) to {x -k,k), so we have hu = (^^^(fe-,)- Thus, 



2— Q(..-)^ E G:;:,) 



0<k<l<x+2i-j 



X + i \ j X + j 
k 



+ 



/ X -\- i \ ( X + j 



^ \l + i — i J \ k 



Now we replace I by x + 2i — j — I in the first sum and k by x + 2i — j — k in the 
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k+l<x+2i-j ^ ^ ^ ^ 



+ E 

k+l>x+2i-j 



X + i \ / X + j 

1 + 3 - + - 2i 



For fixed values of r — k + l both sums can be simplified further by the Vandermonde 
sum (see e.g. [7, sec. 5.1, (5.27)]), so 



•)2x+i 



r<x+2i—j ^ ' r>x+2i—j ^ 

and finally, after replacement of rhy2x+i+j—r in the first sum, and by 2x+4:i—2j—r 
in the second sum, 

2x + i + j\ -^-^ f2x + i + j 



r>x+2j—i ^ ' r<x+2i—j ^ 

= E C'V^'> P-^) 

x+2i—j<r<x+2j—i 

As is well-known, the square of a Pfaffian equals the determinant of the correspond- 
ing skew-symmetric matrix (see e.g. [15, Prop. 2.2]). The quantity Q{i,j), as given 
by (2.5), has the property Q{i,j) = —Q{j, i), due to our interpretation (2.3) of limits 
of sums. Hence, the square of the Pfaffian in (2.4) equals deto<i<j<n-i {Qihj))^ 
which in view of (2.5) is exactly (2.2a). That the Pfaffian itself is the positive square 
root of the determinant is due to the combinatorial interpretation in item (2) of the 
Theorem. Thus, item (3) is established for even n. 

Now let n be odd. Still, by the proof of (2), the constant term in (1.1) equals the 
number of all families {Pq, Pi, . . . , Pn-i) of nonintersecting lattice paths, where Pi 
runs from (— 2z, i) to some point on the antidiagonal line X1+X2 = x, i — 0,1, . . . , n—1. 
However, to apply Theorem 3.1 of [15] again we have to add a "dummy path" P_i of 
length 0, running from {2x, —x) to {2x, —x), say (cf. the Remark after Theorem 3.1 
in [15]; however, we order all paths after the dummy path). See Figure l.a for the 
location of P_i. We infer that the constant term in (1.1) equals 

Pf (QiiJ)), (2.6) 

— l<i<3<n—l 

where Q{i,j) is the number of all pairs {Pi, Pj) of nonintersecting lattice paths. Pi 
running from (—2i,i) to the line xi + X2 = x if i > 0, P-i running from (2x, —x) to 
X1+X2 = X (hence, to {2x, —x)), and Pj running from (— 2j, j) to the line X1+X2 = x. 
If < i < j < n - 1, then Q{i,j) = Y.x+2i-j<r<x+2j-i C'^V^^) according to the 
computation that led to (2.5). Moreover, we have Q{—l,j) = 2^+-' since a pair 
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{P-i,Pj) is nonintersecting for any path Pj running from (— 2j, j) to xi + X2 = x. 
The latter fact is due to the location of P_i, see Figure l.a. Therefore, the square of 
the Pfaffian in (2.6) equals 

/ : \ 



det 

— l<i,j<n— 1 







: ■<r^ (2x+i+j\ 

_2x+i . 2^ [ r ) 

\^ : x+2i—j<r<x-\-2j—i j 



-1 



(2.7) 



i > 



2, 

i 

into 



J = -1 J > 

We subtract 2 times the {j — l)-st column from the j-th column, j = n — l,n — 
. . . , 2, in this order, and we subtract 2 times the {i — l)-st row from the z-th row, 
= n — 1, n — 2, . . . , 2. Thus, by simple algebra, the determinant in (2.7) is turned 



det 

— l<i,j<n- 




-2^' 



2x 

* 





* 





* 







j = -l j = 



(2x+i+j-l)! (3x+3i+l)(3x+3j+l)(3j-3t) 
(a;+2i-j+l)! {x-\-2j-i-\-l)\ 



i = -l 
z = 

i>l 



{2.1 



By expanding this determinant along the top row, and the resulting determinant along 
the left- most column, we arrive at (2.2b), upon rescaling row and column indices. 

Thus the proof of Theorem 1 is complete. □ 

Remark. Mills, Robbins and Rumsey [10, Theorem 1 + last paragraph of p. 281] 
showed that shifted plane partitions of shape (n — l,n — 2,...,1), where the entries in 
row i are at least n — i and at most n, i = 1, 2, . . . , n — 1, are in bijection with totally 
symmetric self-complementary plane partitions contained in a (2n) x (2n) x (2n) box. 
Hence, by item (2) of Theorem 1, the number (1.1) generalizes the number of these 
plane partitions, to which it reduces for a; = 0. 

The idea that is used in the translation of item (1) into item (2) of Theorem 1 is 
due to Doran [4, Proof of Theorem 4.1], who did this translation for x = 0. However, 
our presentation is modelled after Stembridge's presentation of Doran's idea in [15, 
Proof of Theorem 8.3]. 

3. A two-parameter family of determinants. The goal of this section is to eval- 
uate the determinant in (2.2a). We shall even consider the generalized determinant 



D(a;, y; n) := det 

0<i,j<n—l 



E 



x+2i—j<r<y+2j—i 



X + y + t + J 

r 



(3.1) 



for integral x and y, which reduces to (2.2a) when y — x. In fact, many of our argu- 
ments essentially require this generalization and would not work without it. Recall 
that the sums in (3.1) have to be interpreted according to (2.3). 



THE ELECTRONIC JOURNAL OF COMBINATORICS 4 (1997), #RXX 



9 



The main result of this section, Theorem 2 below, allows to evaluate D{x,y;n) 
when the difference m = y — x is fixed. It is done explicitly for a number of cases 
in the subsequent Corollary 3, including the case m — which gives the evaluation 
of (2.2a) that we are particularly interested in. For the sake of brevity, Theorem 2 
is formulated only ior y > x (i.e., for m > 0). The corresponding result fory<x is 
easily obtained by taking advantage of the fact 

D{x,y;n) = i-irDiy,x;n), (3.2) 

which results from transposing the matrix in (3.1) and using (2.3). 

Theorem 2. Let x,m,n be nonnegative integers with m < n. Then, with the usual 
notation {a)k '■= a{a + 1) ■ ■ ■ {a + k — 1), k > 1, (a)o := 1, of shifted factorials, there 
holds 



2x + m + i + j 
r 



D(x,x + m:n)= det ( > ( 

0<i,j<n-l V ^ \ 

x-\-2i—j<r<x+m+2j—i 

{2x + m + i)l{3x + m + 2i + 2)i{3x + 2m + 2i + 2) 



n 



{x + 2iy. {x + m + 2iy. 

Ln/2J 



^x+LW2|)Hi + m)! - n (2. + 2rm/21+2,+ l).A(-;™,«), (3.3) 



where Pi{x; m, n) is a polynomial in x of degree < [m/2\ . If n is odd and m is even, 
the polynomial Pi{x; m, n) is divisible by (2x + m + n). For fixed m, the polynomial 
Pi{x; m, n) can be computed explicitly by specializing x to — \_{m + n)/2\+t~l/2, t = 
0, 1, . . . , [m/2\, in the identity (3.67). This makes sense since for these specializations 
the determinant in (3.67) reduces to a determinant of size at most 2t + 1, as is 
elaborated in Step 6 of the proof, and since a polynomial of degree < \m/2\ is uniquely 
determined by its values at [m/2\ + 1 distinct points. 

Proof. The proof is divided into several steps. Our strategy is to transform D{x,x + 
m;n) into a multiple of another determinant, namely Db{x,x + m;n), by (3.6), (3.8) 
and (3.10), which is a polynomial in x, then identify as many factors of the new 
determinant as possible (as a polynomial in x), and finally find a bound for the 
degree of the remaining polynomial factor. 

For big parts of the proof we shall write y for x + m. We feel that this makes things 
more transparent. 

Step 1. Equivalent expressions for D{x,y;n). First, in the definition (3.1) of 
D{x,y;n) we subtract 2 times the (j — l)-st column from the j-th column, j = 
n — l,n — 2,...,1, in this order, and we subtract 2 times the {i — l)-st row from the 
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i-th. row, i = n — l,n — 2, By simple algebra we get 

D{x,y;n) 

( 



det 

0<i,i<n-l 



{x+y+jy. (x+2y+3j+l) 

{x-j+iy. {y+2jy. 



1 = 

{x+y+iy {2x+y+3i+l) : jx+y+i+j-iy jy-x+Sj-Si) 

\ {x+2iy {y-i+iy : (x+2i-j+iy{y+2j-i+iy I ^y-^ 

j = j>l (3.4) 

On the other hand, if in the definition (3.1) of D{x, y; n) we subtract 1/2 times the 
{j + l)-st column from the j-th column, j = 0, 1 . . . , n — 2, in this order, and if we 
subtract 1/2 times the {i + l)-st row from the z-th row, i = 0, 1, . . . , n — 2, we get 

D{x,y;n) 



= det 

0<i,j<n-l 



1 {x + fj + i.+j + l)l (y-,r+3j-30 
4 (x+2i-j+2)! (y+2j-i+2)! 
x(2x+y+3i+4)(x+2y+3j+4) 


1 {x+y+i+ny (2x+s/+3i+4) 

2 (a;+2i-n+3)! (y+2n-i-2)! 


1 (x+y+j+ny {x+2y+3j+4.) 


^ ^x+y+2n-2-^ 
x+n—l<r<y+n—l 


2 {x+2n-j-2y {y+2j-n+3y. 



j<n-2 



j = n-l 



i<n-2 
i = n — l 
(3.5) 



Step 2. An equivalent statement of the Theorem. We consider the expression (3.4). 
We take as many common factors out of the i-th row, i = 0, 1, . . . , n — 1, as possible, 
such that the entries become polynomials in x and y. To be precise, we take 

{x + y + i)l {2x + y + 3i + 1) 
{x + 2iy.{y + 2n-i-l)\ 

out of the i-th row, z = 1, 2, . . . , n — 1, and we take 

{x + yy. 

L(a; + y)/2j!(y + 2n-2)! 

out of the 0-th row. Furthermore, we take {x + 2y + 3j + 1) out of the j-th. column, 
J = 1, 2, . . . , n — 1. This gives 



D{x,y;n) 



{x + yy. 



n-l 

n 



{x + y + iy{2x + y + 3i + l){x + 2y + 3i + l) 



lix + y)/2\\iy + 2n-2y 11 

/ : 

S(x,y;n) : 

-{y-i+2)2n-2 

j = 



X det 

0<i,j<n-l 



{x + 2iy.{y + 2n-i-iy. 

{x+y+l)j (x-j + 2)L(j,_a,)/2J+j-l 
X(y + 2j + l)2„-2j-2 

• (x+y+i+l)j-i (x+2i-j+2)j_i 



x(j/+2i-i+2)2„-2i-2 {y-x+3j-3i) j 



z = 
i>\ 
(3.6) 



THE ELECTRONIC JOURNAL OF COMBINATORICS 4 (1997), #RXX 



11 



where S{x, y; n) is given by 

L(y-^)/2j 

S(x,y;n)= ^ {x + r + l)i^y_^y2\-r {V - r + l)2n+r-2 

\iy-x)/2-]-i 

+ (x + r+l) l{y-x)/2} -riy-r + l)2n+r-2. (3.7) 

For convenience, let us denote the determinant in (3.6) by DA{x,y;n). In fact, 
there are more factors that can be taken out of DaIx, y; n) under the restriction that 
the entries of the determinant continue to be polynomials. To this end, we multiply 
the i-th row of Da{x, y; n) by {y + 2n — i = 1, 2, . . . , n — 1, divide the j-th 

column by {y + 2j + l)2n-2j-2, j = 1, 2, . . . , n — 1, and divide the 0-th column by 
{y + l)2n-2. This leads to 

n— 1 n— 1 



J](2/ + 2n-z)i_i J] 



1 



1 



7=1 + + l)2n-2j-2 {y + l)2n-2 
/ 



DA{x,y;n) 



det 

0<i,J<n-l 



V 



T{x,y) 

-(y-i+2)i_i 



{x+y+l)j (x-i+2)L(y_^)/2J+j-i 



{x+y+i+l)j-i {x+2i-j+2)j_i 

x{y+2j-i+2)i_i{y-x+3j-3i) j 



i= 
%>\ 

(3.8) 



J=0 j>l 
where T{x,y) is given by 

L(y-^)/2j 

T{x,y) ^ (a; + r + l)L(y_^)/2j-r(2/-^ + l)r 

r=l 

r(y-x)/2l-l 

+ {x + r + l)i^y_^y2}-riy-r + l)r, (3.9) 

or, if we denote the determinant in (3.8) by DB{x,y;n), 



n-l 



DA{x,y;n) = {y + l)2n-2 ]^(2y + 2i + l)n-i-i ■ DB{x,y;n). 



(3.10) 



A combination of (3.3), (3.8), and (3.10) then implies that Theorem 2 is equivalent 
to the statement: 

With DB{x,y;n) the determinant in (3.8), there holds 

n-l 

Db{x, y;n)^Y[ {{2x + y + 2i + 2)i_i {x + 2y + 2i + 2)i_i) 



i=l 



Ln/2J-1 



X Y[ (2x + 2\(y-x)/2]+2i + l)-Pi(x;y-x,n), (3.11) 



i=0 
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where Pi{x;y — x,n) satisGes the properties that are stated in Theorem 2. 

Recall that y = x + m, where m is a fixed nonnegative integer. In the subsequent 
steps of the proof we are going to establish that (3.11) does not hold only for integral 
X, but holds as a polynomial identity in x. In order to accomplish this, we show in 
Step 3 that the first product on the right-hand side of (3.11) is a factor of Db{x, y; n), 
then we show in Step 4 that the second product on the right-hand side of (3.11) is a 
factor of DB{x,y;n), and finally we show in Step 5 that the degree of DB{x,y;n) is 
at most 2("'~^) -|- [n/2\ + [{y — x)/2\, which implies that the degree of Pi{x;y — x,n) 
is at most \_{y — x)/2\ = [m/2j. Once this is done, the proof of Theorem 2 will be 
complete (except for the statement about Pi{x;y — x, n) for odd n and even m, which 
is proved in Step 4, and the algorithm for computing Pi{x; y — x,n) explicitly, which 
is described in Step 6). 

Step 3. 117=1 {{'2x+y+2i+2)i_i {x+2y+2i+2)i_i) isafactorofDB{x,y;n). Here 
wc consider the auxiliary determinant D B{x,y.,y\n), which arises from DB{x,y;n) 
(the determinant in (3.8)) by replacing each occurence of y by y, except for the entries 
in the 0-th row, where we only partially replace y by y, 



DB{x,y,y; n) 



det 

0<i,i<n-l 



^ T{x,y) \ (x+y+l),(a;-i+2)L(«_,)/2j+,-i ^ 

i (a;+y+i+l),_i (x+2i-i+2),-_i 
y : x(y+2i-i+2)i_i (y-x+3i-3i) J 



1 = 
i>l 



j = j>l (3.12) 

with T{x, y) given by (3.9). Clearly, Db{x, y, y; n) is a polynomial in x and y (recall 
that y = X + m) which agrees with Db{x, y; n) when y = y. We are going to prove 
that 

n-l 

DBix,y,y;n) = JJ {{2x + y + 2i + 2)i^i{x + 2y + 2i + 2)i^i) ■ P2{x,y,y;n), (3.13) 

i=l 

where P2{x,y,y;n) is a polynomial in x and y. Obviously, when we set y = y, this 
implies that Y[7=i ((2a: + |/ + 2z-|-2)i_i {x + 2y + 2i + 2)i_i^ is a factor of DB{x,y;n), 
as desired. 

To prove (3.13), we first consider just one half of this product, nr=/(2^ + y -\-2i-\- 
2)i_i. Let us concentrate on a typical factor {2x + y + 2i + I + 1), 1 < z < n — 1, 
1 < I < i. We claim that for each such factor there is a linear combination of the 
rows that vanishes if the factor vanishes. More precisely, we claim that for any z, / 
with l<z<n — 1, 1 <l <i there holds 

^^'y/^^ (2i - 3g + (i - 2g + / + 1),-; (X + 2g + l)2i-2s 

^ ii-s) {s-l)\ (-X -2i-l + s)i-s 

s=l 

■ (row s of Db{x, y, —2x — 2i — I — 1; n)) = (row i of Db{x, y, —2x — 2i — I — 1; n)). 

(3.14) 



THE ELECTRONIC JOURNAL OF COMBINATORICS 4 (1997), #RXX 



13 



To see this, we have to check 

L(i+0/2j 



E 



{2i -3s + l) {i-2s + l + l)s-i {x + 2s + l)2i-2s 



(^ - s) (s - 1)\ {-X -2i-l + s)i-s 

■ {-2x -2i-l-s + l)s-i = {-2x -3i-l + l)i_i, (3.15) 

which is (3.14) restricted to the 0-th column, and 

^^'y^^^^ {2i -3S + I){i-2S + 1 + l)s-l {X + 2S + l)2i-2s 

^ (i-s) (s-iy. {-x-2i-l + s)i-s 

X {-X -2i - 1 + s)j-i (x + 2s - j + 2)j_i 
X {-2x -2i-l + 2j -s + l)s-i {-Sx -2i-l + 3j -3s -1) 
= (-x-i-l)j-i (x + 2i-j + 2)j_i {-2x-3i-l + 2j + (-3x-l + 3j 

(3.16) 

which is (3.14) restricted to the j-th. column, 1 < j < n — 1. Equivalently, in terms 
of hypergeometric series (cf. the Appendix for the definition of the F-notation), this 
means to check 

^ {l-2i-2l- 2x)i-i{l + 21 + x)2i-2i 
{-2i - x)i-i 

1-f + f.-f + il-f + i-2^:--^>2z + 2Z + 2a; ■ 
-f + f,l-i + Z,i+Z + f,l + / + f 

= {-2x - 3i - I + (3.17) 



X 5F4 



and 

{-X - 2z)j_i 



(-3a; -41 + 3j - 2i - 1) {-2x -21 + 2j -2i + l)/_i 



(-X - 2i)i-i 

, 4 I 2i ^- I 4Z I _ 1 2i , 21 
X{x-j +21 + 2)2i+j-2l-l ■ 6^5 ' 



£i_T_|_H_|_T- 1— 
3' 3 J ^ s ^ ^ 3~3' 
i_|_2i_„-,4/, _2i , 21 



-| + i I - I + I' -1 - 2^ + ^' - ^' 2i - 2j + 2; + 2a: ■ 
l-i + /,l-f + / + f,|-i + Z+f ' , 

= (-3a; - Z + 3j - 5i - 1) (1 - 3i + 2j - / - 2x),_i 

x{-i-l- x)j-i{2 + 2i-j+ x)j-i (3.18) 

Now, the identity (3.17) holds since the 5i'4-series in (3.17) can be summed by Corol- 
lary A5, and the identity (3.18) holds since the gi's-series in (3.18) can be summed 
by Lemma A6. 

The product HIL/l^^ + y + 2i + 2)j_i consists of factors of the form (2a; + y + a), 
4 < a < 3n — 3. Let a be fixed. Then the factor (2a; + y + a) occurs in the product 
n"Ji^(2a; + y + 2i + 2)i_i as many times as there are solutions to the equation 

a^2i + l + l, with l<i<n-l, 1<1 <i. (3.19) 
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For each solution {i, I), we subtract the hnear combination 



{2i -3S + 1) {i-2s + l + l)s-l {X + 2S + l)2^-2 

^ (i-s) {s-iy- {-x-2i-l + s)i 



s=l 

■ (tow s of DB{x,y,y;n)) (3.20) 

of rows of DB{x,yjy;n) from row i of D B{x,y^y]n). Then, by (3.14), all the entries 
in row i of the resulting determinant vanish for y = —2x — 2i — I — 1. Hence, {2x + 
y + 2i + 1 + 1) = {2x + y + a) is a factor of all the entries in row i, for each solution 
(i, /) of (3.19). By taking these factors out of the determinant we obtain 

DB{x,y,y;n) = (2x + y + a)#(^°i"*^°°^ °^ ^^-^^^^ ■ D^^\x,y,y;n), (3.21) 

where D^^\x , y , y; n) is a determinant whose entries are rational functions in x and 
y, the denominators containing factors of the form (x + c) (which come from the 
coefficients in the linear combination (3.20)). Taking the limit a; — > — c in (3.21) then 

reveals that these denominators cancel in the determinant, so that D^^\x,y,y;n) is 
actually a polynomial in x and y. Thus we have shown that each factor of rir=/(2^ + 
y + 2i + 2)i_i divides DB{x,y,y;n) with the right multiplicity, hence the complete 
product divides DB{x,y,y;n). 

The reasoning that nr=i {x + 2y + 2i + 2)i_i is a factor of Db{x, y, y; n) is similar. 
Also here, let us concentrate on a typical factor (x + 2y + 2j + / + 1), 1 < j < n — 1, 
1 < ^ < i- This time we claim that for each such factor there is a linear combination 
of the columns that vanishes if the factor vanishes. More precisely, we claim that for 
any j, I with l<j<n — l^l<l<j there holds 

■ (column s of DB{-2y - 2j - I - l,y,y; n)) 

= (column j of DB{-2y - 2j - I - 1, y, y; n)). (3.22) 

This means to check 

L^^'^/'J {2j-3s + l){j-2s + l + l),.i 

X {-y - 2i - l)s i-2y - 2j - / - s + l)L(y-x)/2j+s-i 

= i-y - 2j - l)j {-2y - 3j - I + 1) l(j,-,)/2J+,-i, (3.23) 
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X {^y -2j-l + i)s-i {-2y - 2j - I + 2i - s + l),_i 

X {y + 2s-i + 2)i_i {3y + 2j + / - 3z + 3s + 1) 

= {-y-2j-l + i)j-i (-2y-3j-/ + 2i + l),_i (y + 2j -i + 2)i_i {3y + 5j + 1 + 

(3.24) 



which is (3.22) restricted to the z-th row, l<z<n — 1. If we plug 

{-y-2j-l)s = 



i-y - 2j - I), 



{-y-2j-l + s) 



into (3.23), we see that (3.23) is equivalent to (3.15) (replace x hj y and i by j). 
Likewise, by plugging 

{-y -2j - I + s)j_s {-y-j-l)i_i 

into (3.24), we see that (3.24) is equivalent to (3.16) (replace x by y and interchange 
i and j). By arguments that are similar to the ones above, it follows that 11^=1^ + 
2y + 2z + 2)i_i divides DB{x,y,y;n). 

Altogether, this implies that Y[7=i ((2a; + y + 2z + 2)i_i(a; + 2y + 2z + 2)j_i) divides 
DB{x,y,y; n), and so, as wc already noted after (3.13), the product nr=i^ ((2a; + y + 
2i + 2)i-i{x + 2y + 2i + 2)j_i) divides Db{x, y; n), as desired. 

Step I U\2o^~\'2x+2 \{y - x)/2] +2i+l) is a factor ofDB{x, y; n). We consider 
(3.5). In the determinant in (3.5) we take 

l{x + y + i + l)l{2x + y + 3i + 4) 
2 {x + 2i + 2)! {y + 2n- 2)! 

out of the z-th row, z = 0, 1, . . . , n — 2, we take 

{x + y + n)l 
{x + 2n-2)l{y + 2n-2)l 

out of the (n — l)-st row, and we take 

^{y + 2j + 3)2n-2,-4 {x + 2y + 3j + 4) 
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out of the j-th column, j = 0, 1, . . . , n — 2. Then we combine with (3.6) and (3.10) 
(recall that DB{x,y;n) is the determinant in (3.6)), and after cancellation we obtain 



DB{x,y;n) = 



X det 

0<jj<n-l 



2n-2 



{x + y + l)r 



{[{X + y)/2\ + l)2n-2-L(j/-x)/2j {y + l)2n-2 







{x+y+i+2)n-i (a;+2i-n+4)„_ 


x(2/+2j-i+3)i(j/-x+3j 


-Si) 


x(y+2n—i—l)i 


— {x+y+n+l)j {x+2n—j- 
x(j/+2j-n+4)„. 


-1 


U{x,y;n) 


j<n-2 




j = n-l 



i<n-2 
i = n — l 
(3.25) 



where 



y-x- 



U{x,y;n)= ^ {x + y + n + 1)^-2 (x + n + r)n_r-i {v + n - r)r+n-i- 

r=0 

The determinant on the right-hand side of (3.25) has polynomial entries. Note that in 
case of the (n — 1, n — l)-entry this is due to n — r — 1 > n — {y — x — l) — l = n — m > 
(recall that y = x + m), the last inequality being an assumption in the statement of 
the Theorem. The product in the numerator of the right-hand side of (3.25) consists 
of factors of the form (x + y + a) — {2x + m + a) with integral a. Some of these factors 
cancel with the denominator, but all factors of the form {2x + 26 + 1), with integral 
b, do not cancel, and so because of (3.25) divide DB{x,y;n). These factors are 



\{m+n)/2']-\m./2']- 

n 

i=0 



(2a; + 2[m/2] + 2i + 1) 



(with m = y — X, of course). Since 

\{m + n)/2] - \m/2] - 1 = | 



[n/2\ n odd, m even 

|_n/2j — 1 otherwise. 



it follows that Hi^^o^'' ^(2^e + 2 [m/2] + 2z + 1) is a factor of Db{x, y; n), and if n is 
odd and m is even {2x + m + n) is an additional factor of Db{x, y; n). 

Summarizing, so far we have shown that the equation (3.11) holds, where Pi{x;y — 
X, n) — Pi{x] m, n) is some polynomial in x, that has (2x + m + n) as a factor in case 
that n is odd and m is even. It remains to show that Pi(x; m, n) is a polynomial in 
X of degree < [m/2j, and to describe how Pi(a;;m, n) can be computed explicitly. 

Step 5. Pi{x;m, n) is a polynomial in x of degree < [m/2\. Here we write x + m 
for y everywhere. We shall prove that Da{x,x + m;n) (which is defined to be the 
determinant in (3.6)) is a polynomial in x of degree at most 2(^) + ("-^) + [n/2\ + 
lm/2\ . By (3.10) this would imply that Db{x, x + m; n) is a polynomial in x of degree 
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at most 2('^~^) + Ln/2J + |_m/2j, and so, by (3.11), that Pi(a;;m,n) is a polynomial 
in X of degree at most [m/2j, as desired. 

Establishing the claimed degree bound for Da{x,x + m;n) is the most delicate 
part of the proof of the Theorem. We need to consider the generalized determinant 

Da{x, z{1), z{2), ...,z{n- 1); n) = DA{n) 

which arises from Da{x,x + m;n) by replacing each occurence of i in row i by an 
indeterminate, z{i) say, z=l,2, ...,n— 1, 



DAix, z{l), z{2), ...,z{n- 1); n) 
/ 

S{x,x + m; n) 
:= det 

0<i,j<n-l 

— {x+m—z{i)+2)2n-2 

j = 



{2x+m+l)j {x-j+2)^^/2i+j-i 

X {x+m+2j + l)2n-2j-2 



\ 



{2x+m+z{i)+l)j-i (x+2z(i)-j+2)j_i 

x{x+m+2j-z{i)+2)2.n-2j-2 {m+3j-3z{i)) j 



i = 
(3.26) 



where 5'(x, x + m; n) is given by (3.7). 

This determinant is a polynomial in -2(1), -^(2), . . . , z{n — 1). We shall prove that 
the degree in x of this determinant is at most 2(2) + ("^2^) + L^/2J + L^/2J, which 
clearly implies our claim upon setting z{%) = i, i = 1,2, ...,n — 1. 

Let us denote the (z,j)-entry of -Da(^) by A{i,j). In the following computation 
we write Sn for the group of all permutations of {0, 1, . . . , n — 1}. By definition of the 
determinant we have 

n-l 

DAin) = J] sgna JJ A{a{j),j), 
creSn i=o 

and after expanding the determinant along the 0-th row, 

n-2 

DA{n) = A{0,0) J2 llA{aU) + l,j + l) 

n—l n—2 

+ Y^{-1YA{0,£) Y1 sgnal[A{a{j) + l,j + x{j>£)), (3.27) 

where x(«4.)=l if A is true and x(«4.)=0 otherwise. Now, by Lemma AlO we know 
that for > 1 we have 



A{i,j)= J2 2^a,MxPz{i)'^, 
P,q>o 



(3.28) 
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where ap^q{j) is a polynomial in j of degree < 2(2n — 3— p — q)+q — 1. It should be 
noted that the range of the sum in (3.28) is actually 

0<p<2n-4, <q <2n-3, p + q <2n-3. (3.29) 

Furthermore, by Lemma A9 we know that for j > 1 we have 

A(0,j) = J]2^73^(j>^ (3.30) 

p>0 

where Pp{j) is a polynomial in j of degree < 2(2n + [m/2j — 3 — p). Also, for i > 1 
let 

A{i, 0) = -(x + m - z(^) + 2)2n-2 - ^P^i f^^-^l) 

p,g>o 

Plugging (3.28) and (3.31) into (3.27), and writing z{i) instead of z{i-\-l) for notational 
convenience, we get 



n-2 n-2 

1o 



DA{n) = AM Yl 2(^)x^°+-+P"-nap.,g.(i + l) sgna ^(^(j)) 

P0v,Pn-2>0 j—0 aeSn-1 j=0 

qo,...,qn-2>0 

n—1 n—2 

+ 5^(-i)^^(o,£) Y 2(^)-Vo+-+^'^-7po,.o n «P.,..(i + x(j > i)) 

£=1 po,---,Pn-2>0 3 = 1 

qo,--,qn-2>0 

n-2 

X sgna JJ^(cT(j))''^ 

o-eSn-i i=o 

n-2 

=^(0,0) Y 2(^)x^°+-+^"-n«^'.-.^.(-^'+i)n<-^^<^ 

0<i,j,<n— 2 

pO,---,Pii-2>0 j=0 
90,---,gn-2>0 

n—1 n—2 

+ Yi-^Ymi) Y 2(^)-vo+-+^"-7po,.o n + xO' > ^)) 

£=1 po,---,Pn-2>0 J = l 

qo,---,qn-2>0 

X det (^(i)'^O- (3.32) 

0<t,j,<n— 2 

The determinants in (3.32) vanish whenever qj^ = qj^ for some ji ^ j2- Hence, in 
the sequel we may assume that the summation indices qo,qi, . . . , qn-2 are pairwise 
distinct, in both terms on the right-hand side of (3.32). In particular, we may assume 
that in the first term the pairs {po, qo), (pi, gi), . . . , {pn-2, qn-2) are pairwise distinct, 
and that in the second term the pairs {pi, qi), (p2, 52)7 ■ • • 7 {Pn-2-, qn-2) are pairwise 
distinct. What we do next is to collect the summands in the inner sums that are 
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indexed by the same set of pairs. So, if in addition we plug (3.30) into (3.32), we 
obtain 

DA{n) = A{0,0) 2(2)a;Po+-+P"-2 

{iP0,Q0),---,iPn-2,qn-2)} 

n-2 

n-l 

+ E E 2(2)x^+^o+-+^"-7po,.oE(-1)'/^pW 

P,PO,qO>0 {{pi,qi),...,{pn-2,qn-2)} ^=1 

n-2 

X E o<.1t-2^""^^)'^"')n«P.a).«.(^^ (3-33) 

where Sn-2 denotes the group of aU permutations of {0, l,...,n — 1} that fix 0. 
Clearly, we have 

,^.det (^(i)«^(^))=sgnT det {z{iy^). (3.34) 
Moreover, there holds 

n—l n—2 
n-l 

= 5^(-l)%(^) ^^.det («^^,,^(j + x(j > £))) 

'—^ l<i,7<n— 2 

= (-1)^-^ det i<n-2\ 

the step from the last line to the next-to-last line being just expansion of the deter- 
minant along the bottom row. Using (3.34) and (3.35) in (3.33) then yields 

:D^(n) = yl(0,0) J] 2(^)a;^'°+-+^'"-^ 

{(PO,go),---,(PK-2,QTi-2)} 

X det {ziiY') det (ap. (j + 1)) 

0<z,j<n-2^ ^ ^ 0<*J<n-2^ ^^'^'^-^ 

+ (-1)'^-' E E 2(^)a;^+^°+-+^'^-7po,.o 

P,Po,Qo>0 {(pi,qi),...,(p„_2,g„-2)} 

x_det (z(z)^0,..det f^^^'^/jf'^ 'ir^V (3-36) 



0<i,i<n-2 l<j,i<n- 1 \ /3p(j) i = n-l 

We treat the two terms on the right-hand side of (3.36) separately. Recall that we 
want to prove that the degree in x of -D^(n) is at most 2(2) + ("^2 ) + L"'/2J + L™/2J • 
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What regards the first term, 

A{0,0) J2 2(^^)xP°+-+P"-^ 

{(PO,9o),---,(Pn-2,gn-2)} 

x^^.det (z{i)'i^) det + 1)), (3.37) 

0<i,j<n— 2 0<i,j<n—2 

we shall prove that the degree in x is actually at most 2(2) + ("2^) + [{n — 1)/2J + 
[m/2j. Equivalently, when disregarding ^(0,0) = S{x,x + m;n), whose degree in x 
is 2n — 2 + [m/2j (see (3.7)), this means to prove that the degree in x of the sum in 

(3.37) is at most 3 (''2^) + [(n - 1)/2J. 

So we have to examine for which indices po, . . . , Pn-2, Qo, ■ ■ ■ , Qn-2 the determinants 
in (3.37) do not vanish. As we already noted, the first determinant does not vanish 
only if the indices qo,qi, . . . , qn-2 are pairwise distinct. So, without loss of generality 
we may assume 

0<qo<qi<---<qn-2. (3.38) 

Turning to the second determinant in (3.37), we observe that because of what 
we know about ap^^q^{j + 1) (cf. the sentence containing (3.28)) each row of this 
determinant is filled with a single polynomial evaluated at 1,2, ...,n — 1. Let M be 
some nonnegative integer. If we assume that among the polynomials o:-pi,qi{j + 1), 
i = 0,l,...,n — 2, there are M + 1 polynomials of degree less or equal M — 1, then 
the determinant will vanish. For, a set of M + 1 polynomials of maximum degree 
M — 1 is linearly dependent. Hence, the rows in the second determinant in (3.37) 
will be linearly dependent, and so the determinant will vanish. Since the degree of 
oipi,qi (j + 1) as a polynomial in j is at most 2(2n — ?> — pi — qi) -\- qi — 1 (again, cf. the 
sentence containing (3.28)), we have that 

the number of integers 2(2n — 3 — — gj) + gj — 1, i = 0, 1, . . . , n — 2, , . 
that are less or equal M — 1 is at most M. \ • ) 

Now the task is to determine the maximal value of po + Pi + ' ' ' + Pn-2 (which is 
the degree in x of the sum in (3.37) that we are interested in), under the conditions 

(3.38) and (3.39), and the additional condition 

< Pi < 2n - 4, < 9i < 2n - 3, Pi + < 2n - 3, (3.40) 

which comes from (3.29). We want to prove that this maximal value is 3 ("^2^) + 
|_(n — 1)/2J. To simplify notation we write 

ei = 2n-Z-pi-qi. (3.41) 

Thus, since 

n-2 n-2 

Y^p, = {n- l)(2n - 3) - ^(?i + e^). 



i=0 i=0 
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we have to prove that the minimal value of 

qo + qi-\ hgn-2 + £o + £iH hen-2, (3.42) 

under the condition (3.38), the condition that 

£. >0, i = 0, l,...,n-2, (3.43) 

(which comes from the right-most inequahty in (3.40) under the substitution (3.41)), 
the condition that 



the number of integers 2si + Qi — 1, i = 0,1, ... ,n — 2, 
that are less or equal M — 1 is at most M, 



(3.44) 



(which is (3.39) under the substitution (3.41)), and the condition 

if go = 0, then £o > 1, (3.45) 

(which comes from (3.40) and (3.41)), is (''2^) + \(n - l)/2]. 

As a first, simple case, we consider go ^ 1- Then, from (3.38) it follows that the 
sum Yl7=o ^* alone is at least (2) = ("^^) + (^ ~ 1), which trivially implies our claim. 
Therefore, from now on we assume that go = 0. Note that this in particular implies 
Eq > 1, because of (3.45). 

Next, we apply (3.44) with M = 2. In particular, since among the first three 
integers 2ei + gi — 1, i = 0, 1,2, only two can be less or equal 1, there must be an 
ii <2 with 2ei-^ + g^^ — 1 > 2. Without loss of generality we choose ii to be minimal 
with this property. Now we apply (3.44) with M — 2ei^ + qi^ ■ Arguing similarly, we 
see that there must be an 12 < 2ei^ + g^^ with 28^^ + q^^ — 1 > 2£j^ + g^^ . Again, 
we choose 12 to be minimal with this property. This continues, until we meet an 
ik ^ + 5ife-i with 2£i^ + g^^ — 1 > n — 2. That such an i^ must be found 

eventually is seen by applying (3.44) with n — 2. 

Let us collect the facts that we have found so far: There exists a sequence ii,i2, ■ ■ ■ , 
ik of integers satisfying 

0<ii<i2<---<ik<n-2 (3.46) 
(this is because of the minimal choice for each of the i/s), 

ii <2, 12 < 2£ii + gii, . . . , ik< '^£ik-i + qik-i-> (3-47) 

and 

2£i, + gi, - 1 > n - 2. (3.48) 

The other inequalities are not needed later. 

Now we turn to the quantity (3.42) that we want to bound from above. We have 

n — 2 n—2 n — 2 / ^\ n— 2 

^5, + ^£, = ^(5,-z)+ r~ l+J^^i- (3-49) 

i=0 i=0 i=0 ^ ^ i=0 
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For convenience, we write for — i in the sequel. Because of (3.38) we have 

qi>0, z = 0, l,...,n-2, (3.50) 

and 

Qi > Qj, for i > j. (3.51) 
For a fixed i let s be maximal such that is < i- Then, because of (3.51), there holds 

qi-i = qi = iqi- qij + (Qis - Qis-i) + H (gia - g^J + g^i 

> (Qi. - Qi.-i) + \-iQi2-Qii) + Qii- 

Using this, (3.50) and (3.48), in (3.49), we obtain 

n— 2 n— 2 / -i \ k 



^qi + ^£i>y ^ J + {n-l-ii)qi^+^{n-l-is){qi,-qi,_^) 

i=0 i=0 ^ ^ s=2 

n-2 ^ 

+ E^^-^H + ^~V'" - (3.52) 

Now, by (3.47) we have for g^^ that 

k 

(lik = iik +ik = qii + ^{Qis - Qis-i) + (3.53) 

s=2 

k 

8=2 

A similar estimation holds for gi^..^, etc. Thus, by iteration we arrive at 

k 

qiu <kqi, +E(/c-s + l)(gj^ - gi^_ J + 2(6^^ + • • • + £u_i) + 



s=2 

Using this inequality in (3.52), we get 

n-2 n-2 

2 2 



i=0 i=0 



+ (n - 1 - ii - ^) gi, + ^(n - 1 - - - — ^^) (g^^ - gi^_ J 
^ s=2 ^ (3.54b) 

n-2 A: 
i=0 s=l 
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By (3.50), (3.51), and since because of (3.46) we have 

k — s + 1 ^ , ^ , , k — s + 1 k — s + 1 ^ /„ 
n-l-is >n-l-{n-2-k + s) --— = --— > 0, (3.55) 

all terms in the line (3.54b) are nonnegative. If i\ = 0, then by (3.43) the line (3.54c) 
is nonnegative. If 1 < ii < 2 (zi cannot be larger because of (3.47)), then occurs 
in the line (3.54c). As we already noted, we have £o > 1 since we are assuming that 
go = in which case (3.45) applies. So, £q — > 0, which in combination with 
(3.43) again implies that the line (3.54c) is nonnegative. 
Hence, we conclude 

n— 2 n— 2 / -i \ -i 

n — 1 \ n — 1 



i=0 i=0 \ / - 

which is what we wanted. 

The reasoning for the second term om the right-hand side of (3.36), 

P,Po,qo>0 {(pi,gi),...,(p„-2,Qn-2)} 

X det (z(i)'i^) det f "^^^'^/.^-^^ i<n-2\ .g^^g. 

0<i,j<n-2^ l<i,j<n-l \ Pp{J) i = n-l J 

is similar, only slightly more complicated. We shall prove that the degree in x in 
(3.56) is at most 2(2) + ^2^) + [n/2} + [m/2j, which by the discussion in the first 
paragraph of Step 5 is what we need. 

So, we have to determine the maximal value of p+po + ■ ■ ■ +Pn-2 such that the de- 
terminants in (3.56) do not vanish. Basically, we would now more or less run through 
the same arguments as before. Differences arise mainly in the considerations concern- 
ing the second determinant (which is slightly different from the second determinant 
in (3.37)). What has to be used here is that Pp{j) is a polynomial in j of degree 
< 2(2n -I- |_m/2j — 3 — p) (see the sentence containing (3.30)). If we make again the 
substitutions 

Si = 2n — 3 — Pi — Qi, z = 1, 2, . . . , n — 2, (3.57) 
and in addition the substitutions 

£0 = 2n - 2 - po - go, (3.58) 

and 

£ = 2n+ [m/2j -3-p, (3.59) 

we obtain eventually the following conditions that are necessary to make these two 
determinants not vanish: There must hold 

< gi < g2 < • • • < Qn-2, and go is distinct from the other g^'s, (3.60) 
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(this is the substitute for (3.38)), 

£i > 0, i = 0, l,...,n-2, and £ > 0, (3.61) 
(this is the substitute for (3.43)), and finally, 

the number of integers in the set {2ei + qi — 1 : i — 1,2, . . . ,n — 2} U {2e} . . 
that are less or equal M — 1 is at most M, \ • ) 

(this is the substitute for (3.44)). Since by the substitutions (3.57)-(3.59) we have 
P+^Pi = 2( 2) +'^[2 ) ^^'^'^^^ m - £-^(gi + £i), 

i=0 \ / \ / ^=0 

the task is to prove that the minimal value of 

qo + qi-\ \-qn-2+S + So + Si-\ |-£n-2, (3.63) 

equals ("-^) + [(n-2)/2]. 

Next in the arguments for the first term on the right-hand side of (3.36) came the 
sequence of applications of (3.44). Hence, now we apply (3.62) repeatedly. Actually, 
there is only one slight change, with the start. Namely, first we apply (3.62) with 
M = 2£ + 1. Since then 2e is already less or equal M — 1, among the first 2e + 1 
integers 2£i + — 1, i — 1, 2, . . . , 2£ + 1, only 2£ can be less or equal 2£. Hence there 
must be an zi < 2£ + 1 with 2£j^ + qi^^ — 1 > 2e + 1. Continuing in the same manner 
as before, we obtain a sequence ii,i2, ■ ■ ■ ,ik of integers satisfying 

1 <ii <i2 < ■■■ <ik <n-2, (3.64) 

ii < 2£ + 1, 12 < 2£i^ + , . . . , ik< '^^ik-i + qik-i ' (3-65) 

and 

2£u + gife - 1 > n - 2. (3.66) 

Now we turn to the quantity (3.63) that we want to bound from above. We want to 
parallel the computation (3.49)-(3.54). However, since by (3.60) the qiS are slightly 
unordered (in comparison with (3.38)), we have to modify the definition of gj. Namely, 
let t be the uniquely determined integer such that qt < qo < qt+i, if existent, or t = 
if go < gi) or t = n — 2 if qn-2 < go- Then we define 

{go - t if i = 0, 
gj-i + l if 1 <i <t 
qi — i if i > t. 

If we modify (3.49) accordingly, 

^gi + ^£, = (go-t) + ^(g^-z + l)+ J] (5. _ ^) + r ~ ) + J]^^, 

i=0 i=0 i=l i=t+l ^ ^ i=0 
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all subsequent steps that lead to (3.54) can be performed without difficulties. (A little 
detail is that in (3.53) the equality qi^ = qi^ + ik has to be replaced by the inequality 
9ifc < qik + ik-) Also, the estimation (3.55) still holds true because of (3.64). Hence, 
when we use the first inequality in (3.65), together with (3.50), (3.51), (3.55), (3.61), 
in (3.54), we obtain 



n — 2 n— 2 / -i \ -i n—2 k <-> , -i 

i=0 i=0 ^ ^ i=0 s=l 



Si 



> 



n — l\ n — 2 



2 ' 



which is what we wanted. 

The proof that the degree of the polynomial Pi{x;m,n) is at most [m/2j is thus 
complete. 

Step 6. An algorithm for the explicit computation of Pi{x;m,n). Also here, we 
write X + m for y everywhere. A combination of (3.11) and (3.25) yields 



Pi{x; m, n) 



2n-2 



L(m+n)/2j-[m/2j 

n (2a; + 2 [m/2\ + 2i) 



2) {x+ [m/2\ + l)2n-2-Lm/2j {x + m + l)2n-2 

n-1 



1 



X 



X det 

0<i,j<n-l 



n (3a; + TO + 2z + 2)i_i (3a; + 2m + 2i + 2)i_i 

^ {2x+m+i+2)j {x+2i-j+3)j ':{2x+m+i+2)n-i (x+2i-n+4)„_^ 
x{x+m+2j-i+3)i (m+3j-3i): x(x+m+2n-i-l) 

-{2x+m+n+l)j {x+2n-j-l)j: 



X {x+m+2j-n+4:)n-i 

j<n-2 



U{x,x + m; n) 
j = n-l 



i<n-2 
i = n — l 
(3.67) 



By Step 5, we know that the degree of Pi{x; m, n) is at most |_m/2j. Hence, if we are 
able to determine the value of Pi{x; m, n) at [m/2j + 1 different specializations, then 
we can compute Pi (x; m, n) explicitly, e.g. by Lagrange interpolation. 

The specializations that we choose are of the form —v — 1/2, where v is some 
nonnegative integer. The first thing to be observed is that if we set x = —v — 1/2, 
V integral, in (3.67), then the denominator on the right-hand side of (3.67) does not 
vanish. So, everything is well-defined for this type of specialization. 

Next, we observe that for x = —v — 1/2 "usually" (this will be specified in a 
moment) a lot of entries in the determinant in (3.67) will vanish. More precisely, 
since {2x + m + i + 2)j, which is a term in each entry of the determinant except for 
the (n — 1, n — l)-entry, vanishes if i < —2x — m — 2 — 2v — m — 1 and i + j > 
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—2x — m — 1 = 2v — m, the determinant takes on the form 



j — 2v—m — l 

i 



( 



det 

0<i,i<n-l 



V 



* * 

: * 

* •••0 





* 


M 



\ 



i — 2v — m — \ 



(3.68) 



/ 



Obviously, this picture makes sense only if— 1 <2t; — m — l<n — 1, or equivalently, 
if m/2 < V < (m + n)l2. It should be observed that this constraint is met by the 
choices v = [(m + n)/2j , [(m + n)/2j — 1, . . . , [(m + n)/2j — |_m/2j that are suggested 
in the statement of Theorem 2. In particular, for the lower bound this is because of 
the assumption m < n. 

Because of the 0- matrix in the upper-right block of the matrix in (3.68), it follows 
that the determinant in (3.68) equals the product of the determinant of the upper-left 
block times det{M). Since the upper- left block is a triangular matrix, we obtain for 
the determinant in (3.68) an expression of the type 

(product of the elements along the antidiagonal i + j = 2v — m — 1) 

X det (M). 

{m+n—2v) X {m+n—2v) 

In the notation of the statement of the Theorem, i.e., with v = [{m + n)/2\ — t, the 
dimension of det(7W) is {m + n) — 2 |_(m -|- n)/2j -|- 2t, which is less or equal 2t + 1. 

Summarizing, we have seen that for a; = — [(m + n)/2\ + t — 1/2, t = 0, 1, ... , 
[m/2j, the determinant in (3.67) reduces to a (well-defined) multiple of a determinant 
of dimension at most 2t + l. Since we assume m to be some fixed, explicit nonnegative 
integer, and since 2t-|-l < m + 1 (m + 1 being a fixed bound), this determinant can be 
computed explicitly (at least in principle), and so also the explicit value of Pi{x; m, n) 
at X = — [{m + n)/2\+t — l/2, t = 0, 1, . . . , [m/2j. So, the value of Pi{x; m,n) can be 
computed explicitly for [m/2 J -|- 1 distinct specializations, which suflBces to compute 
Pi{x;m,n) explicitly by Lagrange interpolation. 

This finishes the proof of Theorem 2. □ 

Wc have used Theorem 2 to evaluate the determinant D(x,x + m;n) for m = 
0, 1, 2, 3, 4. This is the contents of the next Corollary. 

Corollary 3. Let x and n be nonnegative integers. Then the determinant 



D(x, X + m:n) = det 

0<i,3<n-l 



E 

x+2i—j<r<x+m+2j- 



2x + m + i + j 
r 
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for m = equals 



n 



n/2-l 

i\ {2x + i)l {3x + 2i + 2yf\ Bo 



n-1 ... , , ..2x n (2a; + 2z + l) 



(a; + 2i)!2 ; (n-1)!! 



n even 



i=0 

n odd 

n "^y "n'((«)'^p^+^'+i)'^) (3.69) 

n odd, 

/or m = 1, n > 1, equals 

Ln/2J-1 

Tt' A! (2a; + z + 1)! (3a; + 2i + 3), (3a; + 2z + 4)A ^^^ + ^^ + 3) 

11 (a; + 2z)!(a; + 2z + l)! J (2 K2j - 1)!! ' 

for m — 2, n >2, equals 

Ln/2J-1 

^i!(2x + z + 2)!(3x + 2z+4),(3x + 2i + 6)i^ ^ (2a; + 2z + 3) 



n 



i \ i=0 



(x + 2i)!(x + 2i + 2)! J (2 [?i/2j - 1)!! 



1 ( (x + n + 1) n even 

X ■ <^ ^ ^ 3.71 

(a; + 1) i (2a; + n + 2) n odd, 



for m = 3, n > 3, equals 

n-1 

n 



Ln/2J-1 

^z!(2a; + z + 3)!(3a; + 2z + 5)i(3a; + 2z + 8)i^ ^ (2a; + 2z + 5) 



i=0 



i \ i=0 



(a; + 2z)!(a; + 2z + 3)! J (2 [n/2j - 1)!! 



X ^ f (a; + 2n + 1) n even 
(a; + 1) ' \ (3a; + 2n + 5) n odd, 



and for m — 4, n > 4, equals 



Ln/2J-1 

^ f il {2x + /+ !)! (3.i: + 2/ + 6). (3.r + 2/ + 10)A E ^^'^ + + 
»i V (a; + 2z)!(a; + 2z + 4)! J (2 [n/2j - 1)!! 

1 f (a^^ + (4n + 3)a; + 2{n'^ + 4n + 1)) n even 

^ {x+ l){x + 2) ' \ (2a; + n + 4)(2a; + 2n + 4) n odd. ^' ' 

□ 



At this point we remark that (3.69) combined with Theorem 1, item (3), (2.2a), 
settles the "n even" case of the Conjecture in the introduction, see Theorem 11. 
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We have computed Pi{x;m,n) for further values of m. Together with the cases 
m = 0, 1, 2, 3, 4 that are displayed in Corollary 3, the results suggest that actually a 
stronger version of Theorem 2 is true. 

Conjecture. Let x, m, n be nonnegative integers with m < n. Then 

\ 1 Z' f2x + 'm + i+jW 

D{x,x + m:n)= det > 

0<i,j<n-l V ^ \ r J J 

n-1 

= n 



i=l 



x+2i—j<r<x+m+2j—i 

i\ (2x + m + i)l {3x + m + 2i + 2)i {3x + 2m + 2i + 2), 



(x + 2i)\ (x + m + 2iy. 



Ln/2J-1 

(2r + m)! ^ (2,r + 2 rm/2l + 2/ + 1) 
{x+M2\)i[x + m)r (2K2J-1)!! ^3(x;m,n), (3.74) 

where P3{x; m, n) is a polynomial in x o/ exact degree |_m/2j . In addition, if the cases 
n even and n odd are considered separately, the coefficient of x^ in P3{x;m,n) is a 
polynomial in n of degree [m/2\ — e with positive integer coefficients. 

Note that P3{x;m,n) = Pi{x;m,n) ■ (2 [n/2\ — l)!!/n"jQ^i! (compare (3.74) and 
(3.3)). 

Possibly, this Conjecture (at least the statement about the degree of Ps{x;m,n)) 
can be proved by examining the considerations in Step 5 and Step 6 of the proof of 
Theorem 2 in more detail. 

4. Another two-parameter family of determinants. The goal of this section is 
to evaluate the determinant in (2.2b). We shall consider the generalized determinant 

X f {x + y + i + j -iy.{y-x + 3j -3i)\ 

L{x,y-n] :— det — ; -t — , 4.1 

^ ^ o<i,j<n-i\ {x + 2i-j + l)\{y + 2j-i + l)\ j' ^ ^ 

for integral x and y. (On the side, we remark that E{x, y; n) would also make sense for 
complex X and y if the factorials are interpreted as the appropriate gamma functions. 
Proposition 4 below, together with its proof, actually holds in this more general 
sense. This applies also to Proposition 5, as long as m is a nonnegative integer, 
to Corollary 6, to Theorems 8 and 9, and their proofs.) £^(x, j/; n) reduces to the 
determinant in (2.2b) when n is replaced by n — 1 and y is set equal to x, apart from 
the factor nr=o^(^^ + 3z + 4)^ that can be taken out of the determinant in (2.2b). 

Ultimately, in Theorem 8 at the end of this section, we shall be able to evaluate 
the determinant E{x, y; n) completely, for independent x and y. This is different 
from the determinant D{x,y;n) of the previous section. But, there is a long way 
to go. The first result of this section. Proposition 4, describes how the determinant 
E{x,y;n) factors for independent x and y, however, leaving one factor undetermined. 
It provides the ground work for the subsequent Proposition 5 that makes it possible to 
evaluate E{x, y; n) when the difference m = y — x is fixed. This is then done explicitly 
for two cases in Corollary 6. This includes the case m — which gives the evaluation 
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of the determinant in (2.2b) that we are particularly interested in. The rest of the 
section is then dedicated to the complete evaluation of the determinant E{x,y;n), 
for independent x and y. This is finally done in Theorem 8. Before, in Lemma 7, 
we collect information about the polynomial factor P4{x,y;n) in the factorization 
(4.2) of E[x, y; n). The proof of Theorem 8 then combines this information with the 
evaluation of E{x, a; + 1; n), which is the second case of Corollary 6. 

Proposition 4. Let x, y, n be nonnegative integers. Then 

{x + y + i + j — 1) \ {y — X + ?>j — ?>i) 



E{x, y; n) = det 



n-l 



o<i,j<n-i \ {x + 2i-j + 1)! {y + 2j-i + 1)! 
{x + y + i - l)\ {2x + y + 2i + l),{x + 2y + 2i + 1)^ 



where P4{x, y; n) is a polynomial in x and y of degree n. 

Proof. Again, the proof is divided into several steps. The strategy is very similar to 
the proof of Theorem 2. First, we transform E{x, x + m;n) into a multiple of another 
determinant, namely EBix,y;n), by (4.3)-(4.5), which is a polynomial in x and y, 
then identify as many factors of the new determinant as possible (as a polynomial in 
X and y), and finally find a bound for the degree of the remaining polynomial factor. 

Step 1. An equivalent statement of the Theorem. We take as many common factors 
out of the i-th row of £'(x, y; n), i = 0, 1, . . . , n — 1, as possible, such that the entries 
become polynomials in x and y. To be precise, we take 

[x -\- y + i — 1)\ 



(x + 2i + l)!(y + 2n-i- 1)! 
out of the i-th row, i = 0, l,...,n— 1. This gives 

E(r V n^ - Tf {x + y + i-l)\ 

^ Al (^ + 2i + l)!(y + 2n-i-l)! 

X det ((x + 2/ + i)j {x + 2i-j + 2)j (y + 2j - i + 2)2n-2j-2 {y - x + 3j - 3i)) . 

0<i,j<n-l 

For convenience, let us denote the determinant in (4.3) by EA{x,y;n). In fact, there 
are more factors that can be taken out of EA{x,y;n) under the restriction that the 
entries of the determinant continue to be polynomials. To this end, we multiply the 
z-th row of Ea{x, y;n) by {y + 2n — i)i, i = 0, 1, . . . , n — 1, and divide the j-th column 
by {y + 2j + 2)2n-2j-2, j = 0, 1, . . . , n - 1. This leads to 

n— 1 n— 1 ^ 

n (y + 2n - i). n (,+2,-+2)„ ■ ^) 

= det {{x + y + i)j{x + 2i-j + 2)j{y + 2j -i + 2)i{y-x + 3j -3i)), 

0<i,j<n-l 
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or, if we denote the determinant in (4.4) by EB{x,y;n), 



n-1 



Ea{x, y;n)^Y[{y + 2i + 2)n-i-i ■ Eb{x, y; n). (4.5) 



i=0 



A combination of (4.2), (4.4), and (4.5) then imphes that Proposition 4 is equivalent 
to the statement: 

With Eb{x, y; n) the determinant in (4.4), there holds 



n-1 



EB{x,y;n) = ((2a; + y + 2i + l)i{x + 2y + 2i + 1)^) • P4ix,y;n), (4.6) 

i=0 

where P^i^x, y; n) is a polynomial in x and y of degree n. 

Step 2. nr=o^ {{2x + y + 2i + l)i{x + 2y + 2i + l),) is a factor of EB{x,y;n) . There 
are not many differences to Step 3 of the proof of Theorem 2. So we shall be brief 
here. 

We first consider just one half of this product, nr=o^(2^ + y + 2i + l)j. Let us 
concentrate on a typical factor {2x + y + 2i + l + l),0<i<n — 1, 0<l<i. We 
claim that for each such factor there is a linear combination of the rows that vanishes 
if the factor vanishes. More precisely, we claim that for any i, I with < i < n — 1, 
< I < i there holds 

^^'y/^^ {2i -3s + l){i-2s + l + l)s-i {x + 2s + 2)2i-28 
^ (i-s) {s-iy. i-x-2i-l + s-l)i_s 

• (row s of Eb{x, —2x — 2i — I — 1] n)) = (row i of Eb{x, —2x — 2i — I — 1] n)). 

(4.7) 

Restricting (4.7) to the j-th column, it is seen that this means to check 

{2i -3S + 1) {i-2s + l + {X + 2S + 2)2i-2s 

^ {i-s) {s-iy. {-x-2i-l + s-l)i^s 

X {-X - 2i - I + s - l)j {x + 2s - j + 2)j 
X (-2x -2i-l + 2j - s + l)s (-3x - 2z - / + 3j - 3s - 1) 
= {-x-i-l- l)j {x + 2i-j + 2)j {-2x -3i-l + 2j + l)i {-3x -l + 3j -5i-l). 

This is easily done by observing that it is equivalent to (3.16). Arguments that are 
similar to those after (3.18) then show that the complete product nr=o^(2^+2/+2^+l)i 
divides £'b (a;, y; n) . 

The reasoning that nr=o (^ + 2y + 2i + l) i is a factor of EB{x,y;n) is similar. 
Also here, let us concentrate on a typical factor (a; + 2y + 2j + / + 1), < j < n — 1, 
< I < j. This time we claim that for each such factor there is a linear combination 



(,-.^ ^37\I (2/ + 2s + 2)2i-2. 
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of the columns that vanishes if the factor vanishes. More precisely, we claim that for 
any j, I with 0<j<n — 1, 0<l<j there holds 

h + 2. + 2)2,_2. 

• (column s of EB{—2y — 2j — I — l,y; n)) 

= (column j of EB{-2y -2j -l-l, y; n)). (4.8) 

Restricting to the i-th row, we see that this means to check 

L(i+0/2j 

x{-y-2j-l + i- l)s i-2y ~ 2j - I + 2i - s + l)s 
X {y + 2s-i + 2)i {3y + 2j + I - 3i + 3s + 1) 
= {-y-2j-l + i-l)j {-2y-3j-l + 2i + l)j {y + 2j-i + 2)i (3y + 5j + 1 -3i + l). 

The observation that this summation is equivalent to (3.24) with y — y establishes 
the claim. Similarly to as before, this eventually shows that the complete product 
nr=o^ {x + 2y + 2i + l)i divides Eb{x, y; n). 

Altogether, this implies that nr=o^ {(^^ + y + 2i + l)i(x + 2y + 2i + l)j) divides 
EB{x,y;n), as desired. 

Step 3. P4{x,y;n) is a polynomial in x and y of degree n. We shall prove that 
Ea{x, y; n) (which is defined to be the determinant in (4.3)) is a polynomial in x and 
y of (total) degree 3(2) +n. By (4.5) this would imply that Eb{x^ y; n) is a polynomial 
in X and y of degree 2(2) + n, and so, by (4.6), that P4(a;, y; n) is a polynomial in x 
and y of degree n, as desired. 

Here we need to consider the generalized determinant 

Ea{x, y, z{0), z{l), ...,z{n- 1); n) = -E^(n) 

which arises from Ea{x, y; n) by replacing each occurence of i in row i by an indeter- 
minate, z{i) say, z = 0, 1, . . . , n — 1, 

EA{x,y,z{0), z{l), ...,z{n-l);n)= EA{n) 

■■= ,^.det ((a; + y + z{i))j {x + 2z{i) - j + 2)j 

0<i,j<n— 1 

■ {y + 2j - z{i) + 2)2n-2,-2 {y-x + 3j- 3z{i))). (4.9) 



This determinant is a polynomial in x, y, z{0), z{l), . . . , z{n — 1). We shall prove that 
the degree in x and y of this determinant is 3(2) + n, which clearly implies our claim 
upon setting z{i) — i, i — 0,1, . . . ,n — 1. 
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Obviously, the total degree of EAin) in x,y,z{Q),z{l),...,z{n — 1) is at most 
4(2) +n. However, actually it is exactly equal to this upper bound, since the monomial 



n-l 



yK2)+^^(0)0^(l)i...^(n-l) 

occurs only in the product of the main diagonal of the determinant with nonzero 
coefficient, and therefore cannot cancel. On the other hand, when z{ii) — -2(^2) foi' 
some ii ^12-, the rows i\ and 12 in EAin) are identical. Hence EA{n) vanishes in this 
case. This shows that the product no<i<j<n-i('^0)~'^(*)) divides EA{n). Therefore, 
the degree in x and y of -E^(n) equals (4(2) + n) — (2) = 3(2) + which is what we 
need. 

This finishes the proof of Proposition 4. □ 

As we already mentioned, it is possible to compute the polynomial P4^{x,y;n) 
explicitly, see Theorem 8. However, we are not yet in the position to do so. First we 
restrict ourselves to the situation that was studied in Section 3 for the determinant 
(3.1), namely when the difference m = y — x is a fixed integer. In this situation, it 
turns out that the degree of P4{x,y;n) — P4{x,x + m;n), now as a polynomial in 
X of course, shrinks significantly. To be precise, the degree can be at most [n/2j. 
We prove this fact in Proposition 5 below. Moreover, we identify several factors of 
P4:{x, X + m;n), which will be of great help in the proof of Corollary 6. 

Proposition 5. Let x, m, n be nonnegative integers with m < n. Then, 

, , / (2x + m + i+j -1)1 (m + 3j -3i) \ 
E(x, x + m;n)= det -f- Vr 

_Yt f C^x + m + i - 1)1 {3x + m + 2i + 1), {3x + 2m + 2i + l)i \ 
~ V (x + 2i + l)\(x + m + 2i + l)\ / 

X (a; + m)Ln/2j-Lm/2j ■ Pf,{x;m,n), (4.10) 
where P^{x] m, n) is a polynomial in x of degree < [m/2\ . 

Proof. Much of the required work has already been done. In particular, if we set 
y — X + m m. (4.2) and compare with (4.10), we obtain 

P^{x,x + m;n) = {x + m)in/2}-[m/2] ■ P-,{x\m,n). (4.11) 

So what remains to prove is that {x + ?n) |^„/2J - [m/2j is a factor of P4{x, x + m; n), 
and that the degree of P4(a;, a; + m; n) is at most Ln/2J. The first fact is established 
in Step 1, the second in Step 2. 

Step 1. {x + m) \n/2\-\_m/2\ is a factor of P4{x,x + m; n). Fix an integer v, 

m + n — 1 / , X 

m<v< . (4.12) 
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We shall show that x + v is a, factor of P^ix, x + m;n), or equivalently, that x = —v 
is a zero of P4{x, x + m;n). The polynomial P4{x, x + m;n) is defined by (4.6), with 
y = X + m, i.e., by 

n-l 

Eb{x, X + m;n) = JJ^ ((3a; + m + 2i + l)i {3x + 2m + 2i + l)j) • P4,{x, x + m;n), 
i=o (4.13) 

where Eb{x, x+m; n) is the determinant in (4.4) with y = x + m. So, we would like to 
set X = —V in (4.13), prove that Eb{—v, —v + m; n) equals 0, that the product on the 
right-hand side of (4.13) is nonzero, and conclude that therefore P4(— —v + m; n) 
must be 0. However, the product on the right-hand side of (4.13) unfortunately 
(usually) is for x = —v. This makes it necessary to cancel first all factors {x -\- v) 
that occur in the product, and only then set x — —v. 

To accomplish this, we have to "generate" these factors on the left-hand side. This 
is done by reading through Step 2 of the proof of Proposition 4 with y = x + m. To 
make this more precise, observe that x + v divides a typical factor 3x + m-\-2i + l 
0<i<n — 1, 0</<i, of the first half of the product in (4.13) if and only if 
3v — m + 2i + I + 1. So, for each solution (i, I) of 

3v = m + 2i + l + l, with 0<z<n-l, < I < i, (4.14) 

we subtract the linear combination 

L(»g/2J _^^^i^-_2s + l + {x + 2s + 2)2^-2s 



a=l 



(i -s) (s - 0! i-x-2i-l + s-l) 



{row s of Eb{x,x + m;n)) (4.15) 



of rows of Eb{x, x + m;n) from row i of Eb{x, x + m;n). Let us denote the resulting 
determinant by Eb{x; m, n). By (4.7), the effect is that (3a:+m+2z+/-|-l) = (3a;-|-3f ) 
(the equality being due to (4.14)), and hence (x + f ), is a factor of each entry of the 
i-th row of Eb{x; m, n), for each solution (z, /) of (4.14). For later use we record that 
the (i, j)-entry of EB{x;m,n), a solution of (4.14), reads 

(2x + m + i)j {x + 2i-j + 2)j (x + m + 2j -i + 2)i (m + 3j - 3i) 

_ ^^'y/'^ (2Z ~3S + I){i-2S + 1 + {X + 2S + 2)2i-2s 

^ (i-s) (s-iy. {-x-2i-l + s-l)i-s 

X {2x + 'm + s)j {x + 2s-j + 2)j [x + m + 2j -s + 2)s (m + 3j - 3s). (4.16) 

Similar considerations concern the second half of the product in (4.13). Omitting the 
details, for each solution of 



3v ^2m + 2j + l + 1, with < j < n - 1, 0<l <j, 



(4.17) 
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we subtract the linear combination 



L(i+0/2j 



{2j-3s + l) (j -2s + l + l)s-i 



E 



{x + m + 2s + 2)2j-2s 



s=l 



■ (column s of Eb{x; m, n)) 



of columns of EB{x;m,n) (we definitely mean EB{x;m,n), and not EB{x;m,n)) 
from column j of Eb{x, x + 'm;n). By (4.8) , each entry of the j-th column of the new 
determinant will have {x + v) as a factor. We remark that entries that were changed 
by a row and column operations will now have (x + v)"^ as a factor. Now we take 
(x + v) out of the i-th row, for each solution (z, /) of (4.14), and we take (x + v) out of 
the j-th column, for each solution {j, I) of (4.17). We denote the resulting determinant 
by Eb{x; m, n). Thus, we have 



Eb{x, a; + m; n) = (x + ^;)#M"tions of (4.14))+#(solutions of (4.17))£;^(^. 



Plugging this into (4.13), we see that now all factors {x-\-v) can be cancelled on both 
sides, so that we obtain 



for some C{x; m, n) that does not vanish for x = —v. Hence, if we are able to prove 
that EBi—v;m,n) = 0, it would follow that P4(—v,—v + m;n) = 0, which is what 
we want to establish. 

So we are left with showing that EBi—v^m^n) = 0. This will be implied by the 
following two claims: The matrix of which Eb{—v; 171,11) is the determinant has a 
block form (sec (4.18)), where 

Claim 1. the upper-right block, consisting of the entries that are in one of the rows 
0,1, . . . ,2v — m and one of the columns 2v — m + l,2v — m + 2, . . . ,n — 1, is a zero 
matrix, and where 

Claim 2. the determinant of the upper-left block, J\f, consisting of the entries that 
are in one of the rows 0,1, . . . ,2v — m and one of the columns 0,1, . . . ,2v — m, equals 



Eb{x; m, n) = C{x; m, n) P^i^x, x + m; n). 



0. 



j = 2v—m 







i = 2v — m 



(4.18) 



For, the determinant of a block matrix of the form (4.18) equals the product of the 
determinants of the upper-left block and the lower-right block, the first determinant 
being equal to by Claim 2. 
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Claim 1 is most obvious for all the entries that did not change in the transition 
from Eb{x,x + m;n) to EB{x;m,n). For, the (z,j)-entry of Eb{x,x + m;n), by its 
definition in (4.4), is 

{-2v + m + i)j {-V + 2i-j + 2)j {-v + m + 2j - i + 2)i (m + 3j - 3i). (4.19) 

Clearly, if < i < 2t; — m and 2w — m + 1 < j < n — 1, we have {—2v + m + i)j = 0, 
and so the complete expression in (4.19) is 0. 

On the other hand, let us consider an (z,j)-entry of EB{x;m,n) that changed in 
the transition from Eb{x,x + m; n) to EB{x;m,n). First we want to know, where 
such an entry could be located. If it changed under a row operation, then (i, I) is a 
solution of (4.14), for some I. By (4.14) we have 

m + 2i + l<m + 2i + l + l = 3v and 3v = m + 2i + I + 1 < m + 3i, 

and so, 

V <i< . (4.20) 

3 2 

If the (i, j)-entry changed under a column operation, then {j, I) is a solution of (4.17), 
for some /. Similar arguments then give, using (4.17), that 

2m . 3v — 2m — 1 / , x 

v- — <j< . (4.21) 

In particular we have j < 2v — m, so an (z, j)-entry that is located in the upper-right 
block, which we are currently interested in, did not change under a column operation. 

But it could have changed under a row operation. Such an (i, j)-entry is given by 
(4.16) divided by (x + v). (Recall that (4.16) was the expression for an (i,j)-entry 
that changed under a row operation before we factored {x + v) out of the i-th row.) 
Thus, it can be written as 



{2x + m + i)2 



v—m—i-\-l 



{x + v) 



{2x + 2^-1- l)i+j_2v+m-i {x + 2i- j + 2)j 

^0- ■^o^( ^^'^'^ -3s + l)ii-2s + l + 
[X + m + 2j - I + 2)i [m + 3j - 3i) - 2^ _ 



(i-s) (s-iy. 

(2a; + m + s)i-8 V'X + 2t> + l)jj^a_2v+m-i 



{—X — 2i — I + s — 

{x + 2s-j + 2)j {x + m + 2j -s + 2)^ (m + 3j -3s)]. (4.22) 



We have to show that (4.22) vanishes for x — > —v. Because of the denominators, it 
is not even evident that (4.22) is well-defined when x — > —v. However, by (4.20) we 
have 2v — m — i > (v — m + l)/2 > 0, the last inequality being due to our assumption 
V >m. Hence, 

{2x + m + i)2v-m-i+l /o I I O 

■ = {2x + m + l)2v-m-i • 2, 

[x + V) 
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and so the first term in (4.22) is well-defined when x — > —v. Furthermore, the 
denominator in the sum in (4.22) (neglecting the terms that do not depend on x) 
when X —v becomes 



{v-2i-l + s- l)i-s ^ {v - 2i - I + s - 1) ■ ■ ■ {v - i - I - 2). (4.23) 



By (4.14) and (4.20) we have v — i — l — 2 = —2v + i + m — l < ^{—v + m — 3) < 0, again 
the last inequality being due to our assumption v > m. Therefore, all the terms in 
(4.23) are nonzero, which means that the denominator in the sum in (4.22) is nonzero 
when X — > —v. Hence, (4.22) is well-defined for x — > —v. To demonstrate that it 
actually vanishes for x — > —v, we show that the second term in (4.22) (the term in 
big parentheses) equals ior x = —v. 

To see this, set x = —v, and by (4.14) replace Z by 3v — 2z — m — 1 in the sum 
(4.22), and then convert it into hypergeometric notation, to obtain 



2 (3 + 6z + 3j + 4m - 9^;) (l),+,_2^-2 

X {-4v + 2i + 2j + 2m + 3)sv-2i-m-i{^v - 4i - j - 2m)_6„+6i+i+2m+2 



X 6F5 



-2i - j - ^ + 3t;, i - 2i - ^ + 2^;, -1 - f - f + f , 
-1 - 2i - j - ^ + 3v, - 2i - ^ + 2v, -3i - m + 3v 



- f + ^,-l-2i + j + v,-2-2i-2j -2m + 4:v 
-2z-i-m+f,i-2z-i-m+f ' . 



(4.24) 



The 6-^5-series can be summed by means of Lemma A6. Then, after simplification, 
(4.24) becomes 



{i + j- 2v + m- 1)! {-V + 2i-j + 2)j {-v + m + 2j - i + 2)i {m + 3j - 3i), 



which is exactly the first term in big parentheses in (4.22) for x = —v. Therefore, the 
term in big parentheses in (4.22) vanishes for x = —v. This settles Claim 1. 

Next we turn to Claim 2. We have to prove that the determinant of the matrix J\f, 
consisting of the entries of Eb{—v; m, n) that are in one of the rows 0, 1, . . . , 2v — m 
and one of the columns 0, 1, ... , 2v—m (recall (4.18)), equals 0. We do this by locating 
enough zeros in the matrix J\f. 

We concentrate on the entries that did not change in the transition from Eb{x, x + 
m; n) to Eb{x; m, n). For the location of the various regions in the matrix M that we 
are going to describe, always consult Figure 2 which gives a rough sketch. 
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3 

l! 



V — TYl + i — 2 




\ 






3 ~ 2 








/— 


N = 








II 
















III 




IV 



3 = 
2v — m 

i = 



i + j = 2v — m 



i=\v-f]-2 + xim = 0i3)) 



i = 2u — m 



Figure 2 

By earlier considerations, an (z,_7)-entry did not change if i is outside the range 
(4.20), i.e.. 



m' 






— 1 or 







< z < 

and if j is outside the range (4.21), i.e., 
0<J < 



3f — m — 1 



+ 1 < i < n- 1, 



(4.25) 



< 


2m 






— 1 or 







3v — 2m — 1 



+ 1 <j <n-l. 



(4.26) 



As we already noted, such an (i, j)-entry is given by (4.19). The first term in (4.19) 
vanishes if and only if 



i <2v — m and i + j > 2v — m. 
The second term in (4.19) vanishes if and only if 



(4.27) 



v-1 



< i < 



v + j-2 



(4.28) 



The third term in (4.19) vanishes if and only if 



V — m — 1 



<j< 



V — m + i — 2 



(4.29) 



Finally, the fourth term in (4.19) vanishes if and only if 



m 



m = (mod 3) and * = J + — 



(4.30) 



THE ELECTRONIC JOURNAL OF COMBINATORICS 4 (1997), #RXX 



38 



Now we claim that in the following four regions of N all the entries are 0, except 
for the case v = m = 1, which we treat separately. Again, to get an idea of the 
location of these regions, consult Figure 2. 

Region I: All (z, j)-entries with 



'v-l 




m~ 




< i < 




2 







-2 + x(m = (mod 3)) 

, V — m — 1 
and 



< 7 < 



2m' 



1, (4.31) 



where again x{A)=l if A is true and x{A)=0 otherwise. 
Region II: All (z, j)-entries with 



- 2 + x(to = (mod 3)) 

3v — 2m — 1 



'v-l 




m~ 




< i < 




2 




"-3- 



and 



Region III: All (z, j)-entries with 
3f — m — 1 



-\-l<i<2v — m and 

Region IV: All (i, j) -entries with 
3v — m — 1 

+ l<i<2v — m and 



V — m — 1 



+ l<j<2v-m. (4.32) 



< i < 



2m 



1. (4.33) 



3v — 2m — 1 



+ l<j<2v-m. (4.34) 



Instantly we observe that all four regions satisfy (4.25) and (4.26). So, all the 
entries in these regions arc given by (4.19). Hence, to verify that all these entries are 
we have to show that for each entry one of (4.27)-(4.30) is true. Of course, we treat 
the four regions separately. 

ad Region I. First let i < \v — m/3] — 2. In case that i < j -\- m/3, we have 



z + j+f \v-Tf\-2 + j + 



^-f + i-2+J + f _ 



V + ?■ - 2 1 

- + 77- 



• < — ■ 3 < r 3J — - ' J ' z < 

- 2 ~ 2 - 2 2 3 

Combined with (4.31), this implies that (4.28) is satisfied. On the other hand, in case 
that i > j + m/3, or equivalently. 



m 1 

we have, using the last inequality in (4.31), 



(4.35) 



< 



z + i;-^ + |-l-f y-m + i-2 



+ 3- 
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Combined with (4.31), this imphes that (4.29) is satisfied, unless j = {v — m+i — l)/2. 
But if we plug this into (4.35), we obtain i > v — m/3 — 1/3, a contradiction to our 
assumption i<[v — m/3]— 2. 

Collecting our results so far, we have seen that if m = 1,2 (mod 3), then each 
(i,j)-entry in region I satisfies (4.28) or (4.29). If m = (mod 3), region I also 
contains entries from row i = v — m/3 — 1. First let j < v — (2m)/3 — 2. Then it is 
immediate that (4.29) is satisfied. If j = v — (2m)/3 — 1, then (4.30) is satisfied. This 
shows that if m = (mod 3) then an (z, j)-entry in region I satisfies (4.28), (4.29), or 
(4.30). 

ad Region II. Here, by (4.32), we have 



i + j > 



'v - 1 






+ 


2 



3v — 2m — 1 



+ 1 = 2v — m. 



Hence, (4.27) is satisfied, except when i = \{v — l)/2] and j = [{3v — 2m — 1)/2J +1. 
But in that case there holds (4.28), apart from a few exceptional cases. For, if f ^ 0, 2 
(and V >0 oi course) then 



v-1 



< 



By the assumption f > m it follows that 

v+{[ ^''-f'-^ \+l)-2 



< 



(4.36) 



which is nothing but (4.28) with the current choices of i and j. Thus, (4.28) is satisfied 
except when v — m — 0, orv — 2 and m = 0, 1, 2. (There are no more cases because 

V > m.) Starting from the back, the case ?; = m = 2 does not bother us, since in 
that case region II is empty (there is no i satisfying (4.32)). By inspection, it is seen 
that (4.36), and hence (4.28), also holds v = 2 and m = or 1. Finally, in case 

V = m = we have i = \{v — l)/2] = and j = [{3v — 2m — 1)/2J +1 = 0. Hence, 
(4.30) is satisfied. 

ad Region III. By (4.33) we have 



i + j > 



3v 



m 



1 



+ 1 + 



m 



— 2v — m. 



Hence again, (4.27) is satisfied, except when i — L(3f — m — 1)/2J + 1 and j = 
\{v — m — l)/2]. In that case there holds (4.29), apart from a few special cases. For, 
if is a positive integer, then 



w — 1 



< 



w 
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Setting w = V — m in this inequality we obtain for v > m and v >1 the inequahty 



m 



1 



< 



- m + ([ 



3d— m— 1 



J+l)-2 



(4.37) 



This is exactly (4.29) with the current choices for i and j. Thus, (4.29) is satisfied 
except when v = m or v = 0. (Recall that there are no more cases because of 

V > m.) But (4.37), and hence (4.29), holds in more cases. Namely, by inspection, if 

V = m, then (4.37) holds for v > 2. So, the only cases in which (4.37) is not true are 

V = m = and v = m = 1. The case v = m = does not bother us, since in that 
case region III is empty (there is no j satisfying (4.33)). The case v = m = 1 is the 
exceptional case that is treated separately. 

ad Region IV. By (4.34) we have 



i + j > 



3v — m — 1 



+ 1 + 



3^; — 2m — 1 



+ l>Sv \-l>2v-m, 

2 



the last inequality being again due to the assumption v > m. Hence, (4.27) is satisfied. 

Consequently, if we are not in the case f = m = 1, then the rows \{v — l)/2] , . . . , 
\v — m/3] — 2 + x(m = (mod 3)), l{3v — m — l)/2j+l,...,2t' — m are rows with ze- 
ros in columns \{v — m — l)/2] , . . . ,\v — (2m)/3] — 1, l{3v — 2m — 1)/2J +1, . . . , 2v — 
m. These are 



m 



1 + x{m = (mod 3)) 



1 



+ 2v 



m 



3v — m — 1 



rows, containing possibly nontrivial entries in only 



V — m — 1 



+ 



3v — 2m — 1 





2m 







+ 1 



columns. By simple algebra, the difference between (4.38) and (4.39) equals 
m+ -— + — — +x("^ = (mod 3)). 



(4.38) 



(4.39) 



(4.40) 



As is easily verified, the expression (4.40) equals 1 always. So we have found + 1 
rows (with N the expression in (4.39)) that actually live in M-^ (R denoting the set 
of real numbers). Hence, they must be linearly dependent. This implies that the 
determinant of M must be 0. 

Finally we settle the case v = m = 1. The matrix M then is a 2 x 2 matrix (cf. 
Figure 2) in which column 1 vanishes. For, z = and J = 1 satisfy (4.25), (4.26), and 
(4.28), while i = 1 and j = 1 satisfy (4.25), (4.26), and (4.27). Hence, det(A/') = 0. 

Altogether, this establishes that P4^{x,x + m;n) vanishes for x — —v, m < v < 
(n + m — l)/2, so that (x + m)\_(^n_rn+i)/2i is a factor of P4{x , x + m; n) . Since 







2m 




+ 




~ 3" 


3 



n 



m + 1 



l_f J ~ I_tJ + ^ ^ ^^^^ 

Lf J - Lf J otherwise. 
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it follows that (x + m) \n/2\-\m/2\ is a factor of P4(x, x + m; n), as desired. 

Step 2. The degree of P4{x, x + min) is at most [n/2\. Fortunately, this was, 
implicitly, already proved in Step 5 of the proof of Theorem 2. To see this, we 
examine parts of Step 5 in more detail and relate them to P^ix, x + m]n). 

First we go back to (3.37). In what follows after (3.37), it is shown that the degree 
in X of the polynomial (3.37) is at most 2{f) + C^'^) + [{n - 1)/2J + Lm/2J. Now, a 
closer look at the manipulations before (3.37) unfolds that (3.37) equals the first term 
on the right-hand side of (3.27). However, this first term on the right-hand side of 
(3.27) is nothing else but the (0, 0)-entry of the determinant in (3.26) times the minor 
of the same determinant consisting of rows 1, 2, . . . , n — 1, and columns 1, 2, . . . , n — 1. 
This minor is 

det ((2a; + m + zii) + l)j_i {x + 2z{i) -j + 2),_i 

l<i,j<n— 1 

■ {x + m + 2j - z{i) + 2)2n-2j-2 {m + 3j - ^z{i))) 
= det ({2x + m + z{i+l) + 1),- (x + 2z{i + 1) - j + 1),- 

0<i,j<n—2 

■{x + m + 2j- z{i + 1) + 4)2„-2i-4 (m + 3j + 3 - 3z{i + 1))) . (4.41) 

The (0, 0)-entry of the determinant in (3.27) is S{x, x + m;n), which by its definition 
(3.7) is a polynomial in x of degree 2n — 2 + [m/2j (as is also remarked after (3.37)). 
So, the degree in x of the determinant in (4.41) is at most 3("~^) + |_(n — 1)/2J. Now 
it should be noted that upon replacing nbyn+1, x by x — 1, and upon setting 
z{i + 1) = z + 1, i = 0, 1, ... ,n — 1, the right-hand side of (4.41) turns into the 
determinant in (4.3) with y = x + m, which by definition is Ea{x, x m:n). Hence, 
the degree in x of Ea{x, x + m;n) is at most 3(2) + 1^/2] , and so, by (4.5) and (4.6), 
the degree in x of P4(x, x + m;n) is at most [n/2\. 

This finishes the proof of Proposition 5. □ 

In the next step, we use Proposition 5 to evaluate the determinant E{x, x + m;n) 
for m = and m = 1. The case m = is the one that we need for the evaluation of 
the determinant in (2.2b), the case m = 1 is needed for the evaluation of E{x,y;n), 
for independent x and y, in the proof of Theorem 8. 

Corollary 6. Let x and n be nonnegative integers. Then the determinant 

i2x + in + /■ + J - 1)! (ni + 3j - 3/) 



E(x,x + m:n)— det , , „. . 
^ ^ o<ij<n-i\{x + 2i- J + l)\{x + m + 2j + 

for m = equals 

nl 'prV i!(2x + i-l)!(3a; + 2i + l)f \ 
(n/2)! 11 V {x + 2i + l)P j-Wn/2 

' i=0 



n even 



n odd 



n odd, 
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and for m — 1, n > 1, equals 



n\ / z!(2x + 0!(3.r + 2/ + 2).(3.r + 2/ + 3), ^ 

11 ^TT^TXTuTTT^TX^Ti •(^ + l)Ln/2j- (4.43) 



L^/2J ! fJ-Q \ {x + 2i + 1)! {x + 2i + 2)! 



Proof. By Proposition 5 we know exactly how E{x,x + m; n) factors, except for the 
polynomial P5{x;m,n). However, also by Proposition 5, for m = 0, 1 the degree of 
P^ix; m, n) is at most 0. Hence, P^ix; m, n) is a constant for m = and for m = 1. 
A combination of (4.6), with y = x + m, and (4.11) yields 



^ n— 1 

Psix; m, n) = - — ■ ^ TT 



{x + m) |_„/2j - [m/2j iM; (3x + m + 2z + 1)^ (3x + 2m + 2z + l)j 



X 



i=0 

det i(2x + m + i),- (x + 2i - j + 2),- (a; + m + 2j - i + 2)j (m + 3j - 3i)) . 

Thus, the (constant) value of ^5(0:; m, n) can be determined, by finding an appropriate 
special value for x, which allows to evaluate the determinant in (4.44). 

We choose x = ~ \ {n + m)/2j + 1/2, m = 0, 1. With this choice for x, the denom- 
inator on the right-hand side of (4.44) does not vanish. So, everything is well-defined 
for this specialization. In addition, since {2x -\- m -\- which is a term in each 
entry of the determinant, vanishes ifz < —2x — m = 2 [{n + m)/2\ — m — 1 and 
i + j > —2x — m + 1 = 2 L(n -|- m)/2j — m, for m = and m — 1 the determinant 
takes on the form (3.68), where the submatrix Ai is empty or a 1 x 1 matrix. Therefore 
the determinant can be easily evaluated for this specialization. This gives P5(x; m, n) 
for m = and m = 1. Substitution of these values for P5{x] m, n) into (4.10) yields 
the expressions (4.42) and (4.43). □ 

The evaluation for the special case m = a; = is implicitly in [2] . (It is equivalent 
to the determinant evaluation for det{v{n)) in Section 4 of [2].) 

At this point we remark that (4.42) combined with Theorem 1, item (3), (2.2b), 
settles the "n odd" case of the Conjecture in the Introduction, see Theorem 11. 

To be able to evaluate the determinant E{x,y]n) of (4.1) completely, for inde- 
pendent X and y, we need one more auxiliary result. It locates several zeros of the 
polynomial factor P4(a;, y; n) of £^(a;, j/; n) (recall (4.2)). 

Lemma 7. Ifu, v are nonnegative integers with u+v < n—1, then P4{—u, —v; n) = 0, 
with P4{x,y;n) the polynomial in (4.2). 

Proof. Let u, v be nonnegative integers with u + v < n — 1. The polynomial P4{x, y; n) 
is defined by (4.6), 



n-1 

EB{x,y;n) = ((2a; + y + 2z + 1)^ (a; + 2y + 2z + 1)^) • P4{x,y;n), (4.45) 

i=0 
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where EB{x,y;n) is the determinant in (4.4). What we would hke to do is to set 
X = —u and y = —v in (4.45), prove that Eb{—u, —v;n) equals 0, that the product 
on the right-hand side of (4.45) is nonzero, and conclude that therefore P4(— w, —v; n) 
must be 0. However, the product on the right-hand side of (4.45) unfortunately 
(usually) is for x = —u and y — —v. So we are in exactly the same situation as in 
Step 1 of Proposition 5. The specialization of P4(x, y] n) that we are considering here 
is very different, though. Curiously enough, the arguments of Step 1 of Proposition 5 
can still be used here, word by word, with suitable replacements of parameters. To get 
convinced that this is indeed the case, it will suffice to do the very beginning. Soon 
it will become clear that everything runs in parallel with Step 1 of Proposition 5. 

To begin with, we set y = —v in (4.45). Before setting x = —u, we have to cancel 
all factors of the form x + u that occur in the product on the right-hand side of (4.45). 
To accomplish this, we have to "generate" these factors on the left-hand side. Here, 
this is done by reading through Step 2 of the proof of Proposition 4 with y = —v. To 
make this more precise, observe that x + u divides a typical factor 2x — v + 2i + l + l, 
0<i<n — 1, 0<l<i, of the first half of the product in (4.45) if and only if 
2u = —V + 2i + I + 1. Therefore, if we recall (4.7), for each solution (i, /) of 

2u^ -v + 2i + l + l, with 0<i<n-l, 0<l <i, (4.46) 

we subtract the linear combination 

^^'y/^^ {2i -3S + 1) {i-2s + l + l)s-l {X + 2S + 2)2^-2s 

^ (i-s) {s-iy. {-x-2i-l + s-l)i-s 

■ (row s of Eb{x, -v; n)) (4.47) 

of rows of EB{x,—v;n) from row i of EB{x,—v;ri). By (4.7), the effect is that 

(2x — V + 2i + I + 1) = 2{x + u) (the equality being due to (4.46)), is a factor of each 
entry of the z-th row of the new determinant, for each solution (z, /) of (4.46). 

Now it should be observed that (4.46) is exactly equivalent to (4.14) with the re- 
placements f — > u and m ^ u — v, while (4.47) is exactly (4.15) with the replacements 
m ^ y — x and y — > — and the determinant E{x, —v; n) that we are considering here 
is exactly the determinant E{x, x + m; n) that is considered in Step 1 of Proposition 5, 
with the same replacements. This observation makes it apparent that similar replace- 
ments in the rest of Step 1 of Proposition 5 will produce a valid proof of Lemma 7. 
In particular, in (4.17), in the statements of Claim 1 and Claim 2, in (4.20), (4.21), 
Figure 2, (4.25)-(4.40), the replacements v — > tt and m ^ u — v yield what we need 
here. We leave the details to the reader. □ 



Now we are in the position to prove the promised full evaluation of the determinant 
E{x,y;n). 
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Theorem 8. Let x, y, n be nonnegative integers. Then 

{x + y + i + j — 1)1 {y — X + 3j — 3i) 



E(x, y; n) — det 



o<i,j<n-i \ {x + 2i- j + l)l{y + 2j + 

n-l 

n 

i=0 



i\{x + y + i-l)l{2x + y + 2i + l)i{x + 2y + 2i + 1), 
{x + 2i + iy.{y + 2i + iy. 

Proof. Obviously, the Theorem is equivalent to the assertion that with 

(n-l \ / \ 

n^')-E(-l)'( j(^)'^(2/)n-fe (4.49) 
i=0 ^ fc=0 ^ ^ 

there holds P4{x,y;n) — PQ{x,y;n), where P4{x,y;n) is the polynomial in (4.2). 
For the proof of this assertion we check the following properties for PQ{x,y;n): 

(1) ^6(^5 y, ^) is a polynomial in x and y of (total) degree n. 

(2) Pq{—u, —v; n) = for all nonnegative integers u and v with u + v < n — 1. 

(3) P6{y,x;n) = {-irPe{x,y;n). 



(4) PQ{x,X + l■n)^-^^\^l[i\j■ix + l)^r^/2i■ 



i=0 

It should be noted that all these properties are also satisfied by P4{x,y;n). This is 
because of Proposition 4 for (1), because of Lemma 7 for (2), because of 

E{x,y;n) = {-l)''E{y,x;n) (4.50) 

(if combined with (4.2)) for (3) (identity (4.50) results from transposing the matrix 
in (4.1)), and because of (4.43) (if combined with (4.2)) for (4). Since we also show 
that 

(5) The conditions (l)-(4) determine a polynomial in x and y uniquely, 
the assertion follows. 

ad (1). This is obvious from the definition (4.49). 

ad (2). We have {—u)k = ior k > u. Hence, if k > u the corresponding summand 
in the sum in (4.49) vanishes for x = —u and y = —v. Now let k < u. Because of 
u + V < n — 1 ii follows that k < n — v, or equivalently, n — k > v. But this implies 
{—v)n-k = 0. Therefore also any summand with k < u vanishes for x = —u and 
y = —V. Thus, Pq{—u., —v; n) = 0, as desired. 

ad (3). This is obvious from the definition (4.49). 

ad (4). Setting y = a; + 1 in (4.49), we get 



ni=o ^' fc=o 



PQ{x,x + l]n) ^^i^fn 

rn-l 
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or in hypergeometric notation (cf . the Appendix for the definition of the F-notation) , 



Pq{x, X + l;n) 



Next we use the contiguous relation 



—n,x 
—n — X 



-1 



a, b 



— 2F1 



a,b+l 



az 



a + 1,6+ 1 
c+1 



(4.51) 



to obtain 
P6(x,x + 1; n) 



{X + l)n 2F1 



— n, 1 + X 
—n — X 



-1 



+ 



n 



n + X 



iFi 



1 — n, 1 + x _ 
1 — n — X ' 



-1 



To the 2-P'i-series we apply the quadratic transformation (sec [11, (3.2)]) 



iFi 



a, b 
1 + a-b 



{l + z)-\F^ 



a 1 I a 
2 ' 2 2 . 



Az 



l + a-6' (1 + ^)2 



(4.52) 



This gives 

nn—l ■■ 
i=0 



{x + l)n lim (1 + ZY2F1 

Z—f — l 



n 1 n Ay 
■2'2 2 . 



-n-x ' (1 + 2;)^ 



+ 



n 



n + X 



(1 + ^) 



n-l 



2-fl 



1 n -1 n 

2 2 ' 2 
1-n-x '(1 + ^)^ 



t. 1^ 



Now, when performing the limit, only one term survives on the right-hand side, either 
in the first 2-^1 -series or in the second, depending on whether n is odd or even. After 
simplification, it is seen that both cases result in 



Pg{Xj X + l;n) nl 

n - Jo ^! ^ 



+ l)Ln/2j, 



which is what we want. 

ad (5). Let Q{x,y) be a polynomial in x and y satisfying conditions (l)-(4). 
Because of (1), Q{x,y) can be written in the form 



(4.53) 



i,j>0 
i+j<n 



with uniquely determined coefficients ttij. Now, in (4.53) we set x = and y = —v, 
< V < n — 1. Because of (2), we obtain = Yl^=o'^oj From this system 
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of equations we get aoj = for < j < n — 1. Similarly, by using (2) with x = 
— 1, —2, . . . , — (n — 1), we get a^- = whenever i + j < n — 1. 
Thus, Q{x, y) can be written in the form 

n 

y) = ^bk {x)k {y)n-k, (4.54) 

where we set bk '■= ak,n-k- 

Now we apply (3). Since the coefficients bk in the expansion (4.54) are uniquely 
determined, we get bk = {—l)'^bn-k, and so 

Ln/2J \n/2]~l 

Q{x,y) = ^ bk{x)k{y)n-k+ bn-k {x)n-k {y)k 

k=0 k=0 

Ln/2J rV2l-l 

= ^ h {x)k {y)n-k + (-!)"■ ^ h {x)n-k {y)k- 

k=0 k=0 

Finally we set y = a; + 1 in this equation and use condition (4). This leads to 

■n-l \ [n/2\ 



n 



[n/2\ 



Y[i\j{x+1) Ln/2J = (X + 1) Ln/2J ^ bk {x)k {x + ln/2\ + l)n-k- [n/2i 

\n/2]-l 

+ (-l)"(a;+l)Ln/2j Yl hx-{x+ln/2\ + l)r,-k-ln/2\-iix+l)k, 

k=0 



and after cancellation, 

■ n-l X ln/2i 



j:bk{x)k{x+[n/2\+l] 



'ln/2]-k 

i=0 ^ k=0 

\n/2] 

+ i-ir Yl bk-i {x)k ■ {x + ln/2\ + l)^n/2]-k- (4.55) 
fc=i 

We distinguish between n being even or odd. First let n be even. It is straight- 
forward to see that the polynomials 

{x)k {x + n/2 + l)„/2-fc, /c = 0, 1, . . . , n/2, 

are linearly independent. Hence, by comparison of coefficients, equation (4.55) is 
equivalent to a system of equations of the form 



^'O = Co, 6l + 6o = Ci, 62 + = C2, . . . , bn/2 + K/2-1 = C^/2, 
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where cq, ci, . . . , c„/2 are certain uniquely determined constants. This system of equa- 
tions has a unique solution, which implies that Q{x,y) is uniquely determined. 
The case of odd n is handled similarly. Here, the polynomials 

{x)k {x + {n + l)/2)(n+i)/2-fe, /c = 0, 1, . . . , (n + l)/2, 

are linearly independent. Hence, equation (4.55) is equivalent to a system of equations 
of the form 

bo = Cq, bi + bo = c[, 62 + 61 = C2, . . . , 

b{n-l)/2 + &(n-3)/2 = c[n-l)/2i ^n-l)/2 = c[n+l)/2 

where Cg, c'^, . . . , c'^rL+i)/2 certain uniquely determined constants. Again, this sys- 
tem of equations has a unique solution, which implies that Q{x,y) is uniquely deter- 
mined also in this case. 

This completes the proof of the Theorem. □ 

Now we can also say explicitly what the polynomial factor P5{x; m, n) of E[x, x + 
m; n) in Proposition 5 is. 

Theorem 9. Let x,m,n be nonnegative integers with m <n. Then 

, ^ / {2x + m + i + j -l)\{m + 3j -3i) \ 

E(x,x + m;n)= det 7^ — — — ^7 — — — f- 

^ ^ o<i,j<n-i\{x + 2i- J + l)\{x + m + 2j -i + l)\J 



n-l 

n 



n 



i\{2x + m + i - iy.{3x + m + 2i + l)j (3a; + 2m + 2i + 1), ' 
[n/2\\ ^J^^ V {x + 2i + l)\{x + m + 2i + l)\ 

X {X + m)L„/2j-Lm/2j • E + ,^^) 

X (Ln/2J -/c + l)fe(x+ rm/21 + K2j)L^/2j_fc, (4.56) 
again with x(.4)=i if A is true and x{A)=0 otherwise. 

Proof. We put y = x + min Theorem 8. Comparison of (4.48) and (4.56) then reveals 
that we have to show 

= E (2, + ^^i, odd)) <^"/2J -k+iu(.+ r™/2i + L"/2J)lw.j-.^ 



Actually, this was already done for the special case m = 1 when we checked condition 
(4) in the proof of Theorem 8. Therefore we have to generalize what we did there. 
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First we write the left-hand side of (4.57) in hypergeometric notation (cf. the 
Appendix for the definition of the F-notation), 



{x + m)n ■ 2F1 



— n, X 
1 — n — X — 



Iteration of the contiguous relation (4.51) then turns this expression into 



fc>0 ^ ^ ^ 



—n + k,x + m 
1 + k — n — X — 



-1 



Now we can again apply the quadratic transformation (4.52) to obtain 
''m\ {x + m)n {-n)k 



k>0 



k){l — n — X — m)k 



1 



X lim (1 + ^)"-'=2i^i 



'2~'~2' 2 ^ 2 ^ 2 . 



1 + k — n — X — m '(1 + 2;)^ 



and when expanding the 2-P'i-series according to its definition and simplifying a little, 



{x + m)n{-n)k+2i (-1) 



Elm\s;~y {X + m)n { — 
\k) ^ il-n-x- 

fc>0 ^ ^ £>0 ^ 



m) 



k+i 



lim (1 + z) 

z^ — l 



n-k-2(. 



Now, the limit is nonzero only if n and k have the same parity and if £ = (n — k)/ 2, and 
in that case it is 1. By substituting 2k + x{n is odd) for k and by little manipulation 
we arrive finally at the right-hand side of (4.57). Thus, (4.57) is established, and 
therefore the Theorem. □ 

In Section 3 we formulated a Conjecture about the "extra" polynomial factor 
Pi{x; m, n) that occurs in the evaluation of the determinant D{x, x + m;n) in Theo- 
rem 2. An analogous result seems to hold for the "extra" polynomial factor P^[x;m,n) 
of E{x, X + m;n) as given in Proposition 5, which was identified as 



Lr,/2J 



n '0 ■ ^ + x!^ i« odd)) ( L"/2J - * + 1). + lra/2] + Ln/2J ) ,„,.j 

i, — fc ^ 

by Theorem 9. 

Conjecture. Let x, 171,11 be nonnegative integers with m <n. Then the polynomial 



k>0 



m 

2k + x(n is 



is odc^))^^""/^^ + ^""/^^ + K2j)Lm/2j-fe> (4-58) 



a polynomial in x of exact degree \mj2\, satisfies: If the cases n even and n odd are 
considered separately, its coefficient of x^ is a polynomial in n of degree [m/2\ — e 
with positive integer coefficients. 
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5. A related determinant identity. In this section we derive a determinant iden- 
tity that is somewhat related to the determinant identities of the previous sections (see 
the paragraph after the proof of Theorem 10 for an account of this relationship). Spe- 
cial cases of this identity appeared previously in the paper [2] of Andrews and Burge, 
also in connection with the enumeration of totally symmetric self-complementary 
plane partitions. 

In [2, sec. 4], Andrews and Burge show that the determinants in (2.2) for a; = 
(which give the enumeration of totally symmetric self-complementary plane parti- 
tions) can be transformed by elementary row and column operations into the deter- 
minant 



and in Theorem 2 of their paper (see also [3, Theorem 3]) provide an evaluation even 
for 



(We changed the notation of [2] slightly. In particular, we replaced x hj x — 2.) 
Then they observe that the determinant (5.1) reduces to the determinant (5.2) with 
X = 2 and with n replaced by n — 1, and thus provide another proof of the totally 
symmetric self-complementary plane partitions conjecture. However, there is even 
a two-parameter generalization of (5.2), (namely the determinant in (5.4) below), 
that can be evaluated. This two-parameter generalization is the subject of our next 
theorem. We formulate it only for integral x and y. But in fact, with a generalized 
definition of factorials and binomials (cf. [7, sec. 5.5, (5.96), (5.100)]; see also the 
remarks after (4.1)), Theorem 10, together with its proof, would also hold for complex 
X and y. 

Theorem 10. Let x, y, n be nonnegative integers. Then there holds 




(5.1) 




(5.2) 




(5.3) 



or equivalently, 



det ( 



X + y + i + j 
y + 2j -i 




x+y+i+j— 1 
, y + 2j-i-l 




0<i,j<n-l \ 



n-1 



i\{x + y + i-l)\ (2.x + y + 2i), {x + 2y + 2i)i+i 
{x + 2i)\ {y + 2i)l 



n 



(5.4) 



Proof. The equivalence of (5.3) and (5.4) is obvious from the simple fact 



fx + y + i+j\ /x + y + i+ j-l\ _ {x + y + i + j - 1)1 {x + 2y + 3j) 
V y + 2j-i y + 23-1-1 ) ~ {x + 2i-3)\{y + 23-i)\ 
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We are going to prove (5.3). Our procedure is very similar to the preceding proofs 
of Theorems 2 and 4, only that things are much simpler here. Actually, in the research 
process it was the other way round. This proof was found first and provided (some 
of) the inspiration for the later proofs of Theorems 2 and 4. 

Step 1. An equivalent statement of the Theorem. We take as many common factors 
out of the i-th row of the determinant in (5.3), i = 0, 1, . . . , n — 1, as possible, such 
that the entries become polynomials in x and y. Thus we obtain 

n— 1 / . , \ 1 

■Q (.r + y + /-i)! 



LJ- (a; + 2z)!(y + 2n-z-2)! 



X det {{x + y + i)j{x + 2i-j + l)j{y + 2j-i + l)2n-2j-2)- 

0<i,j<n—l 

Comparing with (5.3), we see that (5.3) is equivalent to 

det {{x + y + i)j {x + 2i-j + l)j {y + 2j -i + l)2n-2j-2) 

J<i,j<n— 1 

n-1 

Yl (z! {y + 2i + l)n-i-i {2x + y + 2i)i {x + 2y + 2i)i) . (5.5) 



0<i,_ _ 

n-1 



i=0 



Let us denote the determinant in (5.5) by Ps{x,y;n). 

We are going to establish (5.5), and thus (5.3), by showing in Steps 2 and 3 that 
the right-hand side of (5.5) divides Psix, y; n) as a polynomial in x and y, by showing 
in Step 4 that the (total) degree in x and y of Psix, y; n) is 3(2) , so that Ps{x^ y; n) is 
a constant multiple of the right-hand side of (5.5), and by showing that this constant 
equals 1, also in Step 4. 

Step 2. rir=o^ (y + 2^ + l)n-i-i is a factor of P%{x, y; n). We multiply the z-th row 
of Ps{x, y;n), which is the determinant in (5.5), by {y + 2n — i — l)i, i — 0, 1, . . . , n — 1, 
and divide the j-th column by {y + 2j + l)2n-2j-2, j = 0,1, . . . ,n — 1. This leads to 

n— 1 ^ n — 1 

^»(-. = n fa + 2n-.-l). n + 2J + ihn-2j-, 

i=0 j=Q 

X det ({x + y + i)j {x + 2i-j + l)j {y + 2j-i + 1),) 

0<i,j<n—l 

n-1 

= TT (y + 2i + l)„_^_i • det {{x + y + i)j{x + 2i-j + l)j{y + 2j-i + l)i). 

0<i,j<n— 1 

i=0 

Since the determinant in the last line is a polynomial in x and y, we infer that 
117^0 (y + 2i + l)n-i-i divides Psix, y; n). 

Step 3. nr=o^ ((2^ + y + {x + 2y + 2i)i) is a factor of Ps{x, y; n). We proceed 
in the spirit of Step 3 of the proof of Theorem 2. So it is not necessary to provide all 
the details. The basic ideas will suffice. 
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„ (2i- Zs + (i - 2. + t + !)._, (X + 2. + 1).. 

^ (r — s\ (s — ]\\ ( — rr — Oi — ] -\- - _ ^- 



First, let us concentrate on a typical factor {2x+y + 2i + l), < i < n — 1, < I < i, 
of the first half of the product, nr=o^(2^ + + claim that for each such factor 

there is a linear combination of the rows that vanishes if the factor vanishes. More 
precisely, we claim that for any i, I with 0<i<n — l,0<l<i there holds 

L^^^/'J (2^-3. + 0(^-2. + ^ + l)._, + 2. + . g ■ , , ^ 

^ (i-s) {s-l)l {-x-2i-l + s),.} ^^^"^ ' 

■ (row s of Ps{x, —2x — 2i — I; n)) = (row i of Psix, —2x — 2i — I; n)). 
Restricting to the j-th column, it is seen that this means to check 

L(i+0/2j 

(i-s) {s-iy. {-x-2i-l + s)i_, 

X {-x-2i-l + s)j {x + 2s-j + l)j {-2x -2i-l + 2j -s + l)2n-2j-2 

= (-X - i- l)j (x + 2i- j + l)j (-2x -3i-l + 2j + l)2n-2j-2- 

This is easily done by observing that it is equivalent to (3.15) with x replaced by 
X — j. Arguments that are similar to those after (3.18) then show that the complete 
product nr=o^(2a:; + y + divides Psix, y; n). 

The reasoning that nr=ro^(^ + 2j/+2z)j is a factor of Pgix, y; n) is similar. Also here, 
let us concentrate on a typical factor (x + 2|/ + 2j + /), < j < n — 1, < / < j. This 
time we claim that for each such factor there is a linear combination of the columns 
that vanishes if the factor vanishes. More precisely, we claim that for any j, / with 
< j < n — 1, < Z < j there holds 

^^'i^^^ {2j -3s + l){j-2s + l + l)s-i , , . , 

> T-. ^ 7 jr. (column s of P8[-2y -2j - I, y; n)) 

^ [J-s) (s - /)! 

= (column j of Pgi-^y - 2j - I, y; n)). 
Restricting to the i-th row, we see that this means to check 

L(i+0/2j 



E 



(2j -3s + l) (j -2s + 1 + 



(j -s) {s- 1)1 

X {-y -2j-l + i)s {-2y - 2j - I + 2i - s + l)s {y + 2s - i + l)2n-2s-2 
= {-y -2j-l + i)j {-2y -3j-l + 2i + 1),- {y + 2j-i + \)in-ij-i- (5.6) 

The observation that this summation is equivalent to (3.23) with y replaced by a; + 2 
and y replaced by ?/ — z establishes the claim. Similarly to as before, this eventually 
shows that the complete product nr=o^(-^ + 2?/ + 2?')^ divides P%{x^ y\ n). 

Altogether, this implies that 11^=0^ ((2a; + y + 2i)j(a; + 2j/ + 2i)j) divides P%{x^y\Vb)^ 
as desired. 
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Step 4- Psix, y; n) is a polynomial in x and y of degree 8(2), and the evaluation of 
the multiplicative constant. We consider the generahzed determinant 

P8{x,y,z{0),z{l),...,z{n- l);n) = Ps{x,y\n) 

which arises from Ps,{x., y; n) by replacing each occurence of i in row i by an indeter- 
minate, z{i) say, z = 0, 1, . . . , n — 1, 

Ps{x, y, z{0), z{l), ...,z{n-l);n) = Pgin) 

= det ({x + y + zii))j {x + 2zii) -j + 1),- {y + 2j - zii) + l)2n-2,-2) • 

0<t,3<n—l 

This determinant is a polynomial m x,y, z{0), z{l), . . . , z{n — 1) of (total) degree at 
most 4Q). 

When z{ii) = ^(^2) for some ii ^ ^2, the rows i\ and ^2 in P%{p) are identical. 
Hence P%{vb) vanishes in this case. This shows that the product no<i<i<n-i(-^0) ~ 
z{%)) divides P%{n). Moreover, the argument in the second half of Step 3 shows that 
also XS^ZQ{x-\-2y-\-2i)i divides P%{n)., just replace % by z{%) in (5.6). Thus we obtain 
that 

n-1 

^8(n) = n - -^W) n + 2y + 2z)i • Q{x, y, 2;(0), 2;(1), . . . , 2;(n - 1); n), 

0<i<j<n-l i=0 

(5.7) 

where Q{x, y, z{0), z{l), . . . , z{n — 1); n) is a polynomial in x, y, z{0), ^(1), . . . , z{n — 1) 
of total degree at most 4(^) - 2(^) = 2(^). By comparing coefficients of 

/(S)^(0)"z(l)^---z(n-l)^-^ 

on both sides of (5.7), it is seen that the coefficient of y^^^) in Q{x, y, z{Q), z{l), . . . , 
z{n — l);n) equals 1. 

Now we set z{i) = i, i = 0, 1, . . . , n — 1, in (5.7). Then P%{n) on the left-hand side 
reduces to P8{x.,y:,n). By Steps 2 and 3 we know that 

n-1 

\[{{y + 2i + l)n-i-i{2x + y + 2i),) (5.8) 

divides Ps{x, y; n). Therefore, by (5.7), it also divides Q{x, 0, 1, . . . , n — 1; n). Since 
the degree in x and y of this factor is 2(2), which at the same time is an upper 
bound for the degree in x and y of Q{x,y,0,l, . . . ,n — l;n), as we saw before, 
(5(x, 0, 1, . . .,n — l;n) is a constant multiple of (5.8). Moreover, the coefficient 
of 2/^(2) in (5.8) equals 1, which we already know to be the coefficient of y^^^) in 
Q{x, y, 0, 1, . . . , n — 1; n). Hence, Q{x, y,0,l, . . . ,n — l-.n) agrees with (5.8), which by 
(5.7) completes the proof of (5.5), and hence of the Theorem. □ 
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It should be noted that the two-parameter determinant in (5.4) carries a strong 
relationship to the determinants D{x,y;n) in (3.1) and E{x,y;n) in (4.1). Namely, 
the (i, j)-entry of the determinant in (5.4), with x replaced by x + l, equals the [i,])- 
entry of D{x,y;n) minus 2 times the — l)-entry of D{x,y]n), while {2x + y + 
3i + l){x + 2y + 3j + 1) times the (i, j)-entry of E{x, y; n) equals the (i, j)-entry of 
the determinant in (5.4) with x replaced by a; + 1 minus 2 times the {i — 1, j)-entry 
of the same determinant. So the determinant in (5.4) is somehow "in between" the 
determinants D{x,y;n) and E{x,y;n). 

6. Constant term identities. In this section we translate some of our determinant 
identities into constant term identities. 

Of course, we start by stating the Conjecture of the Introduction, now as a theorem. 

Theorem 11. Let x and n be nonnegative integers. Then there holds 



Also, both the sum of all n x n minors of the n x (2n — 1) matrix 



(x + n — 1, a; + n — 2, . . . , 1), with entries between and n, where the entries in 
row i are at least n — i, i = 1,2, . . . ,n — 1, equal the right-hand side in (6.1 ). 

Proof. For even n, equation (6.1) follows from a combination of Theorem 1, item 
(3), (2.2a), and (3.69). For odd n, equation (6.1) follows from a combination of 
Theorem 1, item (3), (2.2b), and (4.42) or (4.56) with m = 0. The other claims are 
due to Theorem 1, items (1) and (2), respectively. □ 

Next we translate the determinant identities of Section 4 into constant term iden- 
tities. 

Theorem 12. Let x, y, m, n be nonnegative integers with m < n. Then there holds 



CT 




(6.1) 




number of shifted plane partitions of shape 



n-l 



cT(n((i+^o 



x+y+i — l 



{l + 2z^){z^ + 2){z^-l)z: 



-y-2n+i+l 



) 




0<i<j<n-l 




il{x + y + i-l)l{2x + y + 2i + l),+i {x + 2y + 2i + 1)^+1 
{x + 2i + l)\{y + 2i + l)\ 



) 




n 



(6.2) 
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If y = X + m, with m a fixed nonnegative integer, then the constant term in (6.2) 
equals 

n\ 'pr^ fi\ {2x + m + i- 1)! {3x + m + 2i + l)j+i (3a; + 2m + 2i + l)^+i 



n 



LV2J ! fj^ V (a; + 2z + 1)! (a; + m + 2z + 1)! 



X ([n/2j - A; + (x + [m/2l + Ln/2J) L^/2j-fc. (6.3) 

Proof. It is routine to verify that 

(x + y + z + j - 1)! (2a; + y + 3z + 1) (a; + 2y + 3j + 1) (y - a; + 3j - 3z) 

{x + 2i-j)\{y + 2j-i)\ 

= CT ((1 + zf+y+'+^-\i + 2z){z + 2){z- i)z-y-^^+'-'-). 

Consequently, taking determinants we obtain 

/ (a; + y + i + J - 1)! (2x + y + 3i + 1) (x + 2y + 3j + 1) (y - x + 3j - 3i) \ 
o<i J<n-iV (a; + 2i - j)\ {y + 2j - i)\ J 

(6.4) 

= det (CT ((1 + z,r+y+^+^-\l + 2z,){z, + 2){z, - l)^r'"'''+^"')) 

n-1 

= CT ( n ((1 + z,r+y+^-\i + 2zi){zi + 2){zi - i)zry+'-^) 

i=0 

det {{'-^)')) 

'.i,j<n-l W zf ) ) ) 

= CT ( n ((1 + z,r^y^'-^{\ + 2z,){zi + 2){z, - l)zr^-2"+^+i) 

i=0 

n ((^^ ~ ^•?)(^» + + ^i^i))) > 

0<i,i<n-l 

where we used the Vandermonde determinant identity in the last step. Obviously, the 
last line agrees exactly with the left-hand side of (6.2). Thus, by taking factors that 
depend only on z, respectively only on j, out of the determinant in (6.4) and applying 
Theorems 8 and 9 to the resulting determinant, all the assertions of the Theorem 
follow immediately. □ 



X 



n-1 



Finally, we translate the determinant identity of Section 5 into a constant term 
identity. 
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Theorem 13. Let x,y,n be nonnegative integers. Then there holds 

n-1 



i=0 0<i,j<n-l 
n-1 



n i\{x + y + i-l)\{2x + y + 2i), {x + 2y + 2i)i+i 
,=0 {x + 2iy.{y + 2i)\ • ^ • 



Proof. We observe 

/x + y + i + j\ fx + y + i + j-l\^^. 2^ 2^+^) , 

V?/ + 2J-^;V2/ + 2J-^-ly ^ ^ ^ ^ 

and then proceed in the same way as in the proof of Theorem 12. The reader will 
have no difficulties to ffil in the details. □ 

Appendix 

Here we provide auxiliary results that are needed in the proofs of our Theorems. 

We start by recalling the theorems about nonintersecting lattice paths that we 
need in the proof of Theorem 1. The main theorem of nonintersecting lattice paths 
[6, Cor. 2; 15, Theorem 1.2] is the following. 

Proposition Al. Let Aq, Ai, . . . , An-i and Eq, Ei, . . . , En-i be lattice points with 
the "compatibility" property that, given i < j and k < I, any lattice path from 

Ai to El meets any lattice path from Aj to E^. Then the number of all families 
{Pq, Pi, . . . , Pn-i) of nonintersecting lattice paths, where Pi runs from Ai to Ei, 
i = 0,1, . . . ,n — 1, is given by the determinant 

det (\ViAi^E^)\), 

0<i,j<n— 1 

where \V{A — > E)\ denotes the number of all lattice paths from A to E. 

The second result about nonintersecting lattice paths is Stembridge's enumeration 
[15, Theorem 3.1] of nonintersecting lattice paths when the end points of the lattice 
paths are allowed to vary. 

Proposition A2. Let Aq, Ai, . . . , A^-i be lattice points, and let I = {. . . , Ek, 
E'fe+i,...} be a totally ordered set of lattice points, again with the "compatibility" 

property that, given i < j and k < I, any lattice path from Ai to Ei meets any lattice 
path from Aj to Ej.. Then the number of all families (Pq, Pi, ... , Pn-i) of noninter- 
secting lattice paths, where Pi runs from Ai to some point of I, i — 0,1, . . . ,n — 1, is 
given by the Pfaffian 

0<i<j<n—l 
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where Q{i,j) is the number of all pairs {Pi,Pj) of nonintersecting lattice paths, Pi 
running from Ai to some point of I, and Pj running from Aj to some point of I. 

Next we prove some identities for hypergeometric series. We use the usual hyper- 
geometric notation 

(ai)fe 



s 



ai, . . . , 
bi,...,bs 



= E 

fc=0 



{ar)k k 



kl (bi)k ■ ■ ■ iba)k 



where the shifted factorial (a)^ is given by {a)k '■= a{a + 1) ■ ■ ■ (a + /c — 1), k > 1, 
(a)o := 1, as before. 

To begin with, in Lemma A3 wc quote a result of Andrews and Burge [2, Lemma 1]. 
This 4i<3-summation was derived in [2] from a similar 4F'3-summation due to Bailey. 
We provide an alternative proof here, showing that, in fact, Andrews and Burge's 
summation follows easily from a transformation formula due to Singh. (The above 
mentioned 4F3-summation of Bailey's, as well as Lemma 2 of [2] do also follow from 
Singh's transformation formula.) 

Lemma A3. Let n be a positive integer. Then 



4-^3 



n 1 
■ 2 ' 2 

1 - 



-^,-A,A + B 

^ S 1 , B 
2 ' 2 ^ 2 



{A + B)n ^i-A), 



iB)r 



iB)r 



(A.l) 



Proof. In [12, main theorem], Singh proves the following transformation formula (ac- 
tually a Q-analogue thereof, see also [5, (3.10.13); Appendix (III. 21)]): 



4-^3 



a, b, c, d 



— 3-^2 



2a, 26, c 
^ + a + b,c+ d 



provided both series terminate. 

Now, let first n be odd. In Singh's transformation we choose a = —n/2 + e, 
b= 1/2 - n/2, c = -A, and d = A + B. Thus, we obtain 



l-|-e — n, ^,2 + -2" 



l + e-n,B 



Note that because n is odd both series do indeed terminate. We may express the sum 
on the right-hand side explicitly, 

n-l 



-R + e,l-^,-A,A + B 

4-^3 1 BIB 

l-|-£ — n, -2-,2 + "2 

and then let e tend to zero. This gives 



1 



(-^)fc(l -n)fc(2£-n)fc 
^ {l)k{B)k{l + e-n)k 



E 



fc=0 



4-^3 



n 1 
■ 2 ' 2 



1 S 1 , B , -L 

J- <(., 2 ; 2 ^ 2 



n-l 

E 

k=0 
n 

E 

A;=0 
2F1 



{-A)k {-n)k 

{-A)k{-n)k _ (-A)^(-n), 
{l)k{B)k 



-A, -n 
B 



{l)u{B)n 

(-A)^(-n), 

(l)n(S)n 
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The 2^1 -series can be evaluated by the hypergeometric form of Vandermonde sum- 
matfon (see [13, (1.7.7); Appendix (III.4)]), 



2^1 



a, 



-n 



where n is a nonnegative integer. Applying this, we get 



4-^3 



n 1 
■ 2 ' 2 



1 B 1 + S > -L 

J- "'J o J o o 



{-A)n {-n)r 



which reduces to (A.l) since n is odd and thus (— n)„ — —n\. 

The case that n is even is treated similarly. One would start by choosing a = — n/2, 
b = 1/2 — n/2 + e, c = —A, and d = A + B in Singh's transformation formula. We 
leave the details to the reader. □ 

From the 4F3-summation in Lemma A3 we derive a summation for a 5F4-series. 

Lemma A4. Let n be a positive integer. Then 



sF4 



1 _ 2n _ 
^ 3 ' 
_2n -1 



n 



n 1 

2 ' 2 2 ' 



A,A + B 



B 
2 

- 2n) 



1 



' 2 ' 2 

1{A-B 
2 {2A + B) (1 + B), 



(-A). 



+ 



1 {A + 2B + 2n) (A + B)^ 

2 {2A + B) (1 + B)n 



(A.2) 



Proof. We transform the 5F4-series by the contiguous relation 

5-^4 



1 _ 2n 



Ik 1 

2 ' 2 



f,-A,A + S 



zn -1 
" 3 ' '"'2 

3A (S + n] 



2 ' 
4-^3 



2 

n 1 
2 ' 2 

1 - 



n 



4n (2A + B 

3S (2A + S + f ) 



2,1-A^ + S 

/t, 2 2 ' ~ 2 



-4-^3 



An {2A + S) 

3(5+ ^) (A + S 



n 1 
2 ' 2 

1 - 



- f,-A,A + S 

2 ' 2 ^ 2 



+ 



An {2A + S) 



-4-^3 



n 1 
■ 2 ' 2 

1 - 



-§,-A,l + A + S 

„ 1 + B 1 + B ' 

2 ' 2 ' ' 2 



Now each of the 4F3-series can be summed by means of Lemma 1. Some manipulation 
then leads to (A.2). □ 

The special case that is of particular importance in Step 3 of the proofs of Theo- 
rems 2 and 10, and in Step 2 of the proof of Proposition 4, is ^ = 2n + B. 



Corollary A5. Let n be a positive integer. Then 



5-c^4 



1 - 



2n 



2 ' 



-2n- B,2n + 2B 



_ _n 1 
3 ' 2 ' 2 
_2n i_ 1|B liB 



;1 



1 (2n + 2S), 

2 {l + B)n 



(A.3) 



□ 



Finally we move one step further to a e-^^s -summation. 
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Lemma A6. Let n be a positive integer. Then 

4 I 2n I D 1 2n n 1 n -| 



6-^5 



4 I Zn I D 1 ±21 n _L _ 

3~'~3' ' 3' 2'2 

1 I 2n I o 2n -i 

3 3 ' 3 ' 



- -1 

2 ' 



± -r 2 ' 2 ^ 2 



B,2n + 2B 

B '■> 1 



1 (l + 5n + 35) (2n + 2S), 
2(l + 2n + 35) (2 + S)„ 



(A.4) 



Proof. We use the contiguous relation 

" 4 I 2n I D 1 2n n 3^ n 

3~'~3~'" ' 3' 2'2 2' 



-1 -2n- B,2n + 2B 



3' "''-^^2'2^2 



+ 



4 (n + S) (1 + 2n + 2S) 
(l + 4n + 35)(l + 2n + 35) 

X 5F4 

(l + B)(l + 2n + B) 



1 _ 2n 

3 ' 



f , i - f , -1 - 2n - S, 2 + 2n + 2S 



B 



-2n i_ 3,c 
3' "''-^^2'2^2 



(l + 4n + 35) (l + 2n+35) 



5-^4 



1 _ 2n 

3 ' 



2 ' 2 



2n 1 



n 
2 ' 



-2n-S,2n + 2S 



Each of the 5F4-series can be summed by means of Corollary A5. After little manip- 
ulation we arrive at (A.4). □ 

The next Lemmas provide the means for finding degree l)ounds in Step 5 of the 
proof of Theorem 2 and Step 2 of the proof of Proposition 5. 
As usual, given nonnegative integers n and k, we write 



■ X 



E 

l<ii<---<ife<n 

for the elementary symmetric function of order k in xi, . . . , Xn- In particular, this 
definition implies ek{xi, ^2, . . . , Xn) = if n < /c, since then the defining sum is empty. 
The following Lemma (together with its proof) holds with this understanding of the 
definition of elementary symmetric functions. 

Lemma A7. Let a and n be fixed integers, n > 0. Then, as k varies through the 
nonnegative integers, e/s(a, a + 1, . . . , a + n — 1) is a polynomial in n of degree 2k. 

Proof. By induction on k. The assertion is trivially true for k = 0. If we assume that 
the assertion is true for k, we have 



ek+i{a, a + 1, . . . , o + n - 1) 



J2 ^1^2 
a<ii <---<ife-|-i <a+n— 1 



0<ifc-|-i <n— 1 



(a + ik+i)ek{a, a + 1, . . . , a + i^+i - 1). 

(A5) 



By induction hypothesis, efc(a, a + 1, . . . , a + ik+i — 1) is some polynomial p{ik+i) 
in ik+i of degree 2k. Therefore ek+i{a, a + 1, . . . , a + n — 1), which by (A. 5) is the 
indefinite sum of a polynomial of degree 2A; + 1, is a polynomial of degree 2k + 2 (cf. 
e.g. [8, sec. 32, Example on p. 103]). □ 
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Lemma A8. Let a and p be fixed integers, p > 0. Then, as j varies through the 
nonnegative integers, the coefficient of x^~'^ in {x + a)j is a polynomial in j of degree 
2p. 

Proof. By definition of shifted factorials we have 

i-i 

{x + a) j = JJ^ (x + a + i) 

= '^x^~'^ ep{a, a + 1, . . . ,a + j - 1). 

p>0 

(Note that we need not give an upper bound for the sum, since ep(a, a + 1, . . . , a + 
j — 1) = for p > j, see the paragraph before Lemma A7 that contains the definition 
of elementary symmetric functions). Therefore, by Lemma A7, the coefficient of x^~p 
in (x + a)j is a polynomial in j of degree 2p. □ 

Lemma A9. Let n, m, and p he fixed integers < p < 2n + [to/2J — 3. Then, as j 
varies through the integers, 1 < j <n — 1, the coefficient of x^ in 

{2x + m + l)j {x-j + 2)L„i/2j+j-i {x + m + 2j -\- l)2n-2j-2 

is 2^ times a polynomial in j of degree < 2{2n + [?77,/2j — 3 — p). 
Proof. We have 

{2x + m + l)j = x^~'''^^~''br{j), 

r>0 

where, by Lemma A8, br{j) is a polynomial in j of degree 2r. Similarly, we have 
(x- j + 2)L^/2j+,_i = (-l)L-/2j+.-i(_^_ l^/2J)l^/,j^._, 

s>0 

= J^^LW2J+.--i-.(_i).,^(^-)^ 

s>0 

where, by Lemma A8, Cs{j) is a polynomial in ([m/2j + j — 1) of degree 2s, and as 
such is a polynomial in j of degree 2s. Finally, we have 

{x + m + 2j + l)2n-2j-2 = {-X -m-2n + 2)2n-2j-2 

t>0 

where, by Lemma A8, dt{j) is a polynomial in 2n — 2j — 2 of degree 2t, and as such 
is a polynomial in j of degree 2t. 
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Putting things together we get 

{2x + m+ l)j {x-j + 2)im/2i+j-l {x + m + 2j + l)2n-2j-2 



^^^2n+Lm/2J -3-/0 2.- J] (-l)^+*2-^6,(i) C,(j) ^tO')- 



A;>0 r,s,t>0 

r+s+t=k 



Now, the (finite) range of the inner sum does not depend on j. Hence, by what we 
know about br{j), Cs{j), and dt{j), the inner sum is a polynomial in j of degree at 
most 2r + 2s + 2t = 2k. By replacing A; by 2n + [m/2j — 3 — p we get the assertion 
of the Lemma. □ 

Lemma AlO. Let n, m, p, and q be fixed integers < p < 2n — 4, < q < 2n — 3, 
p + q < 2n — 3. Then, as j varies through the integers, 1 < j < n — 1, the coefficient 
of x^z^ in 

{2x + m + z + l)j-i {x + 2z-j + 2)j_x {x + m + 2j - z + 2)2n-2j-2 {m + 3j-3z) (A.6) 

is 2^ times a polynomial in j of degree < 2{2n — 3— p — q)+q — 1. 
Proof. We have 

{2x + m + z+ l),_i = ^(2a; + zy-^-%r{j) 

r>0 

r>Oh>0 ^ ^ 



where, by Lemma A8, br{j) is a polynomial in j — 1 of degree 2r, and as such a 
polynomial in j of degree 2r. Similarly, we have 



{x + 2z-j + 2),_i = + 2zy-'-'cs{j) 

s>0 

s>OA;>0 ^ ^ 

where, by Lemma A8, Cs{j) is a polynomial in j of degree 2s. Finally, we have 

(x + m + 2j - Z+2)2n-2j-2 {-X + Z-m-2n+ l)2n-2j-2 

= Y,{-x + zf--'^-'-%{j) 

t>0 



^ ^ /2n - 2, - 2 - A ^_,y+i^2n-2,-2-t-i^i^^^j^^ 

t>0 l>0 ^ ^ 
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where, by Lemma A8, dt{j) is a polynomial in 2n — 2j — 2 of degree 2t, and as such 
is a polynomial in j of degree 2t. 
Putting things together we get 

{2x + m + z + {x + 2z - j + 2)j_i {x + m + 2j - z + 2)2n-2j-2 {m + 3j - 3^;) 
= (m + 3j - 3z) J2 ^''-^^ 2^ Yl (-1)'+' 2-^-'--'*+'= 

p,q>0 r+s+t=2n—4:—p—q 
h+k+l=q 

Again, the (finite) range of the inner sum does not depend on j. Hence, by what we 
know about br{j), Cs{j), and dt{j), and since a binomial ("'^°^) is a polynomial in j of 
degree the inner sum is a polynomial inj of degree at most 2r + 2s + 2t + h + k + l = 
2{2n — A — p — q) + q. Combining with (m + 3j — Sz), we see that the coefficient of 
x^z'i in (A. 6) is a polynomial in j of degree at most 2{2n — 3— p — q) + q — 1, as 
desired. □ 
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